Sep 11 13:57:45.923300 ip-10-0-140-144 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 13:57:46.320783 ip-10-0-140-144 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:57:46.320783 ip-10-0-140-144 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 13:57:46.320783 ip-10-0-140-144 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:57:46.320783 ip-10-0-140-144 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 13:57:46.320783 ip-10-0-140-144 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:57:46.321779 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.321709    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 13:57:46.324439 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324425    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:57:46.324439 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324439    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324443    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324448    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324452    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324455    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324458    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324461    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324464    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324466    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324469    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324472    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324474    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324477    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324480    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324482    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324485    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324488    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324491    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324493    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:57:46.324501 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324496    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324499    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324502    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324504    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324507    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324510    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324513    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324516    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324518    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324521    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324523    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324526    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324528    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324531    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324535    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324539    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324541    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324544    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324547    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:57:46.324948 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324550    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324553    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324555    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324558    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324560    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324563    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324565    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324568    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324572    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324576    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324578    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324582    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324584    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324587    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324589    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324592    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324596    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324598    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324601    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:57:46.325470 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324603    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324606    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324609    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324611    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324614    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324617    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324619    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324622    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324624    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324627    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324629    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324631    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324634    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324637    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324639    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324642    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324644    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324647    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324650    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324652    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:57:46.325916 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324655    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324657    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324660    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324662    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324665    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324668    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324670    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324673    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324676    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324679    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324681    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324684    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324687    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324689    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324692    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324695    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324698    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324701    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324703    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324706    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:57:46.326413 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.324708    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325400    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325407    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325410    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325413    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325416    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325419    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325422    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325425    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325428    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325430    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325433    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325436    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325438    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325441    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325443    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325446    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325449    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325451    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325454    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:57:46.326886 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325458    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325460    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325465    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325469    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325472    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325475    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325478    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325482    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325484    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325488    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325492    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325495    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325497    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325500    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325503    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325507    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325510    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325514    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325516    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:57:46.327395 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325524    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325528    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325530    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325533    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325536    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325539    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325542    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325544    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325553    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325555    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325558    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325560    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325563    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325566    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325569    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325571    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325574    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325577    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325580    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325583    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:57:46.327888 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325585    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325588    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325591    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325593    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325596    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325598    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325601    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325604    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325606    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325609    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325611    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325613    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325616    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325624    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325627    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325630    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325632    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325635    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325638    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325640    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:57:46.328389 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325643    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325645    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325648    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325650    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325653    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325656    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325659    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325661    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325664    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325666    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325669    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325671    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325674    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325676    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325679    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325681    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325684    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325686    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325689    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325691    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:57:46.328865 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.325694    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326147    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326154    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326157    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326160    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326163    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326166    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326169    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326172    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326175    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326177    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326181    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326184    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326186    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326189    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326192    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326194    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326197    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326200    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326202    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:57:46.329381 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326205    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326207    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326210    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326212    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326215    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326218    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326220    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326222    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326225    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326227    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326230    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326233    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326235    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326238    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326241    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326243    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326246    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326248    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326251    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:57:46.329884 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326255    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326259    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326262    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326264    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326267    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326270    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326273    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326275    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326278    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326281    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326283    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326286    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326288    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326291    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326293    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326296    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326298    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326303    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326306    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:57:46.330365 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326309    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326311    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326314    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326317    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326319    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326322    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326325    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326327    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326330    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326332    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326335    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326337    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326339    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326342    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326345    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326347    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326350    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326353    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326355    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326358    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:57:46.330830 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326361    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326364    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326366    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326369    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326371    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326374    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326376    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326378    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326381    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326384    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326386    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326388    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326392    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326396    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326399    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326402    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326405    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326410    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326413    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326416    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:57:46.331339 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326418    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.326421    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326500    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326534    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326540    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326544    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326549    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326554    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326558    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326562    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326565    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326568    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326571    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326574    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326577    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326580    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326583    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326586    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326589    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326592    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326596    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326599    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326602    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326605    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 13:57:46.331820 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326608    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326612    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326615    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326618    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326622    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326626    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326629    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326632    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326635    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326638    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326642    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326645    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326648    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326651    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326654    2558 flags.go:64] FLAG: --enable-server="true"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326659    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326663    2558 flags.go:64] FLAG: --event-burst="100"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326666    2558 flags.go:64] FLAG: --event-qps="50"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326669    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326672    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326674    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326678    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326681    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326684    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326687    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 11 13:57:46.332401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326690    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326693    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326696    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326699    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326702    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326704    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326707    2558 flags.go:64] FLAG: --feature-gates=""
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326711    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326714    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326717    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326720    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326723    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326727    2558 flags.go:64] FLAG: --help="false"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326730    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326733    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326736    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326740    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326743    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326746    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326749    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326752    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326755    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326760    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 13:57:46.333113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326763    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326767    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326770    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326772    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326775    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326778    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326781    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326784    2558 flags.go:64] FLAG: --lock-file=""
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326787    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326790    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326793    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326798    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326801    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326803    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326806    2558 flags.go:64] FLAG: --logging-format="text"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326809    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326812    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326816    2558 flags.go:64] FLAG: --manifest-url=""
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326818    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326822    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326825    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326831    2558 flags.go:64] FLAG: --max-pods="110"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326834    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326836    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326839    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 13:57:46.333725 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326842    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326845    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326848    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326851    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326858    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326860    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326865    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326868    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326871    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326876    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326879    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326883    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326886    2558 flags.go:64] FLAG: --port="10250"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326889    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326892    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a3c0c10d291b8b2"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326896    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326899    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326902    2558 flags.go:64] FLAG: --register-node="true"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326905    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326908    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326911    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326914    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326917    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326921    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326923    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 13:57:46.334407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326926    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326929    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326932    2558 flags.go:64] FLAG: --runonce="false"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326936    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326939    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326942    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326945    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326948    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326951    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326954    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326957    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326960    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326963    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326965    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326969    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326972    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326976    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326979    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326984    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326987    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326990    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326994    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.326997    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327000    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327003    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 13:57:46.335005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327005    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327008    2558 flags.go:64] FLAG: --v="2"
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327012    2558 flags.go:64] FLAG: --version="false"
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327016    2558 flags.go:64] FLAG: --vmodule=""
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327019    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327022    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327145    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327149    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327153    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327155    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327159    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327162    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327165    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327168    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327171    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327174    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327176    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327179    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327182    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327184    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327187    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327191    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:57:46.335666 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327193    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327196    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327199    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327202    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327204    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327207    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327209    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327212    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327215    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327217    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327220    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327222    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327225    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327227    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327230    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327233    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327235    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327238    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327240    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327243    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:57:46.336213 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327247    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327250    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327252    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327255    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327259    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327262    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327265    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327268    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327271    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327274    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327277    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327282    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327285    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327289    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327292    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327295    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327298    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327301    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327304    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:57:46.337013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327307    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327318    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327321    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327324    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327326    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327329    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327331    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327334    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327336    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327339    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327341    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327344    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327347    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327351    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327354    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327357    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327360    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327362    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327365    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:57:46.337630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327367    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327371    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327374    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327377    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327380    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327384    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327386    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327389    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327391    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327394    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327397    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327400    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327403    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327406    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327408    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327411    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327413    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327416    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327419    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:57:46.338143 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327421    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327424    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327426    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327429    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327432    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.327434    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.327439    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.334888    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.334902    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.335673    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.335684    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335734    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335739    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335742    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335745    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:57:46.338616 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335748    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335751    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335754    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335757    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335760    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335762    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335765    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335768    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335771    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335773    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335776    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335779    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335783    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335786    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335789    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335793    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335796    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335799    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335801    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335804    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:57:46.338984 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335807    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335811    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335814    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335816    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335819    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335823    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335826    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335829    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335832    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335835    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335837    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335840    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335843    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335846    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335848    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335851    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335853    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335856    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335859    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335861    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:57:46.339495 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335864    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335866    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335869    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335872    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335874    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335877    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335879    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335882    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335884    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335887    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335889    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335892    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335895    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335897    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335901    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335904    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335907    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335910    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335914    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:57:46.339976 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335917    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335919    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335922    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335924    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335927    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335930    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335932    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335934    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335937    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335940    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335942    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335945    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335947    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335950    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335952    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335955    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335957    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335960    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335963    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335965    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:57:46.340463 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335968    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335971    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335973    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335976    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335979    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335981    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335984    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335986    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335990    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335993    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.335998    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.336001    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.336004    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.336007    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.336010    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:57:46.340982 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.336013    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.336018    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338053    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338073    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338079    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338084    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338089    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338094    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338099    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338103    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338108    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338135    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338153    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338163    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338168    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:57:46.341428 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338173    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338202    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338207    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338211    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338215    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338329    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338342    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338346    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338349    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338353    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338356    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338358    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338361    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338365    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338368    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338371    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338374    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338377    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338379    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338404    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:57:46.341797 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338409    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338412    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338417    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338422    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338425    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338428    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338431    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338434    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338438    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338441    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338444    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338447    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338450    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338453    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338456    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338458    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338461    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338465    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338469    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:57:46.342296 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338472    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338474    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338477    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338479    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338482    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338485    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338487    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338490    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338493    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338495    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338498    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338500    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338503    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338506    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338509    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338512    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338515    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338517    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338519    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338522    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:57:46.342749 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338524    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338527    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338530    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338532    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338535    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338537    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338540    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338543    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338546    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338549    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338551    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338554    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338557    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338559    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338562    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338564    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338567    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338569    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338572    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338575    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:57:46.343249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338577    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338580    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338582    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338585    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338587    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338590    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:46.338592    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.338598    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.338703    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 13:57:46.343712 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.342419    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 13:57:46.344139 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.344128    2558 server.go:1019] "Starting client certificate rotation"
Sep 11 13:57:46.344242 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.344227    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:57:46.344275 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.344267    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:57:46.348163 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.348151    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:57:46.349227 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.349213    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:57:46.358931 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.358916    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 11 13:57:46.361848 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.361831    2558 log.go:25] "Validated CRI v1 image API"
Sep 11 13:57:46.362673 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.362659    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 13:57:46.364956 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.364940    2558 fs.go:135] Filesystem UUIDs: map[58b1fd3f-143d-41d4-b12d-a3b3d351c15d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b5d29ddf-996a-4201-8037-d84c97128ce6:/dev/nvme0n1p4]
Sep 11 13:57:46.365008 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.364956    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 13:57:46.370655 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.370556    2558 manager.go:216] Machine: {Timestamp:2026-09-11 13:57:46.369218738 +0000 UTC m=+0.332833704 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096284 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec220f431b363836475c754fbdde3ac3 SystemUUID:ec220f43-1b36-3836-475c-754fbdde3ac3 BootID:b38edbbf-93d2-4e6e-a59b-a2a27ed15d8d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:cd:13:a6:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:cd:13:a6:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:38:99:5b:ce:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 13:57:46.370655 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.370647    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 13:57:46.370795 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.370712    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 13:57:46.373521 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.373499    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 13:57:46.373744 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.373523    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 13:57:46.374409 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.374393    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 13:57:46.374475 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.374414    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 13:57:46.374475 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.374443    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:57:46.375256 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.375235    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:57:46.376063 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.376050    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 13:57:46.377903 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.377885    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:57:46.379588 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379576    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:57:46.379696 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379687    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 13:57:46.379730 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379715    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 13:57:46.379730 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379724    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 13:57:46.379783 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379737    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 13:57:46.379783 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379746    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 11 13:57:46.379783 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.379754    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 13:57:46.380073 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.380061    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:57:46.380119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.380085    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:57:46.383444 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.383428    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 13:57:46.385106 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.385094    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 13:57:46.385152 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.385123    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 13:57:46.386555 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386537    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 13:57:46.386555 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386554    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386560    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386565    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386573    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386578    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386590    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386595    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386601    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386606    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386614    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 13:57:46.386640 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.386619    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 13:57:46.387320 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.387311    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 13:57:46.387320 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.387320    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 13:57:46.390748 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.390725    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 13:57:46.390806 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.390766    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 13:57:46.390846 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.390804    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 13:57:46.391133 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.391121    2558 server.go:1268] "Started kubelet"
Sep 11 13:57:46.391219 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.391175    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 13:57:46.391273 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.391223    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 13:57:46.391313 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.391291    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 13:57:46.391963 ip-10-0-140-144 systemd[1]: Started Kubernetes Kubelet.
Sep 11 13:57:46.393381 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.393362    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 13:57:46.394566 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.394549    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 13:57:46.396125 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.396095    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 13:57:46.398338 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.396376    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-144.ec2.internal.18d448dd176faf48  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-144.ec2.internal,UID:ip-10-0-140-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-144.ec2.internal,},FirstTimestamp:2026-09-11 13:57:46.39109716 +0000 UTC m=+0.354712125,LastTimestamp:2026-09-11 13:57:46.39109716 +0000 UTC m=+0.354712125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-144.ec2.internal,}"
Sep 11 13:57:46.398338 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.397661    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 13:57:46.398338 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.398319    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 13:57:46.398582 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.398541    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:46.399100 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399071    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 13:57:46.399163 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399120    2558 factory.go:55] Registering systemd factory
Sep 11 13:57:46.399163 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399133    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 11 13:57:46.399291 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399254    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 13:57:46.399291 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399266    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 13:57:46.399388 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399342    2558 factory.go:153] Registering CRI-O factory
Sep 11 13:57:46.399388 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399353    2558 factory.go:223] Registration of the crio container factory successfully
Sep 11 13:57:46.399462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399400    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 13:57:46.399462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399416    2558 factory.go:103] Registering Raw factory
Sep 11 13:57:46.399462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.399429    2558 manager.go:1196] Started watching for new ooms in manager
Sep 11 13:57:46.400113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.400098    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 13:57:46.400113 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.400112    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 13:57:46.400415 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.400401    2558 manager.go:349] Starting recovery of all containers
Sep 11 13:57:46.402484 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.402463    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qg4c"
Sep 11 13:57:46.402693 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.402662    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 13:57:46.402795 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.402681    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 13:57:46.411541 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.411526    2558 manager.go:354] Recovery completed
Sep 11 13:57:46.412004 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.411909    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qg4c"
Sep 11 13:57:46.417409 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.417396    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.419097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419081    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.419161 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419109    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.419161 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419119    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.419743 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419729    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 13:57:46.419743 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419743    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 13:57:46.419883 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419762    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:57:46.419883 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.419736    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.420900 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.420888    2558 policy_none.go:49] "None policy: Start"
Sep 11 13:57:46.420947 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.420909    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 13:57:46.420947 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.420918    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 13:57:46.421234 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.421167    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-144.ec2.internal.18d448dd191aeae4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-144.ec2.internal,UID:ip-10-0-140-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-144.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-144.ec2.internal,},FirstTimestamp:2026-09-11 13:57:46.419096292 +0000 UTC m=+0.382711258,LastTimestamp:2026-09-11 13:57:46.419096292 +0000 UTC m=+0.382711258,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-144.ec2.internal,}"
Sep 11 13:57:46.421981 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.421966    2558 policy_none.go:47] "Start"
Sep 11 13:57:46.422877 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.422862    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.422953 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.422885    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.422953 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.422896    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.447555 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.447540    2558 manager.go:337] "Starting Device Plugin manager"
Sep 11 13:57:46.447654 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.447571    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 13:57:46.447654 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.447579    2558 server.go:87] "Starting device plugin registration server"
Sep 11 13:57:46.447891 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.447878    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 13:57:46.447947 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.447892    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 13:57:46.448093 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.448075    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 13:57:46.448261 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.448151    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 13:57:46.448261 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.448160    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 13:57:46.448699 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.448682    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 13:57:46.448774 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.448722    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:46.495850 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.495831    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 13:57:46.497003 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.496984    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 13:57:46.497075 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.497051    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 13:57:46.497075 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.497074    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 13:57:46.497223 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.497145    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 13:57:46.502039 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.502005    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:57:46.548828 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.548810    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.549591 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.549576    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.549655 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.549605    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.549655 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.549615    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.549655 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.549637    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.565907 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.565887    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.565985 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.565911    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-144.ec2.internal\": node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:46.597977 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.597919    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"]
Sep 11 13:57:46.597977 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.597974    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.598720 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.598705    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.598807 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.598734    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.598807 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.598748    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.599954 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.599937    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.600048 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.599968    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.600048 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.599973    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.600713 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6841457108083885ee66eb5ffa556d37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal\" (UID: \"6841457108083885ee66eb5ffa556d37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.600787 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600727    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841457108083885ee66eb5ffa556d37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal\" (UID: \"6841457108083885ee66eb5ffa556d37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.600787 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600752    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49ea5c1f950c94bad71aa5473962c9e7-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.ec2.internal\" (UID: \"49ea5c1f950c94bad71aa5473962c9e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.600897 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600882    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.600937 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600912    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.600937 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600917    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.600937 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600925    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.600937 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600938    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.601094 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.600948    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.602191 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.602174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.602279 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.602207    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:57:46.602928 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.602914    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:57:46.603001 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.602933    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:57:46.603001 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.602945    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:57:46.626553 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.626536    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-144.ec2.internal\" not found" node="ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.630578 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.630562    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-144.ec2.internal\" not found" node="ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.662973 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.661968    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:46.701475 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.701455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6841457108083885ee66eb5ffa556d37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal\" (UID: \"6841457108083885ee66eb5ffa556d37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.701541 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.701479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841457108083885ee66eb5ffa556d37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal\" (UID: \"6841457108083885ee66eb5ffa556d37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.701541 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.701496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49ea5c1f950c94bad71aa5473962c9e7-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.ec2.internal\" (UID: \"49ea5c1f950c94bad71aa5473962c9e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.701541 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.701532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49ea5c1f950c94bad71aa5473962c9e7-config\") pod \"kube-apiserver-proxy-ip-10-0-140-144.ec2.internal\" (UID: \"49ea5c1f950c94bad71aa5473962c9e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.701648 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.701571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6841457108083885ee66eb5ffa556d37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal\" (UID: \"6841457108083885ee66eb5ffa556d37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.701648 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.701571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6841457108083885ee66eb5ffa556d37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal\" (UID: \"6841457108083885ee66eb5ffa556d37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.763054 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.763020    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:46.863589 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.863523    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:46.929183 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.929160    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.934261 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:46.934242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:46.964092 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:46.964063    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:47.064554 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.064520    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:47.165210 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.165191    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-144.ec2.internal\" not found"
Sep 11 13:57:47.226741 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.226714    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:57:47.298805 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.298771    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:47.314069 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.314046    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:57:47.315959 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.315942    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal"
Sep 11 13:57:47.323651 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.323632    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:57:47.343086 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.343066    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 13:57:47.343238 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.343213    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 13:57:47.343285 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.343213    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 13:57:47.380625 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.380601    2558 apiserver.go:52] "Watching apiserver"
Sep 11 13:57:47.397771 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.397750    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 13:57:47.399715 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.399691    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 13:57:47.400101 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.400076    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-bqdgz","kube-system/konnectivity-agent-62pnq","kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal","openshift-dns/node-resolver-6mxh4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal","openshift-multus/multus-additional-cni-plugins-sjdj6","openshift-multus/network-metrics-daemon-th28r","openshift-network-operator/iptables-alerter-2tnsf","openshift-ovn-kubernetes/ovnkube-node-dsrq8","kube-system/global-pull-secret-syncer-qtbqd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5","openshift-cluster-node-tuning-operator/tuned-rl2qm","openshift-image-registry/node-ca-6wjfc","openshift-multus/multus-7hqrb"]
Sep 11 13:57:47.402616 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.402589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:47.402721 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.402673    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:57:47.404788 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.404769    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.406965 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.406942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.407867 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.407693    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 13:57:47.408006 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.407991    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ltq2c\""
Sep 11 13:57:47.408117 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.408001    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 13:57:47.409247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.409225    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lp7gp\""
Sep 11 13:57:47.409355 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.409286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.409561 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.409540    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.409704 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.409685    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.411445 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411423    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.411628 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411597    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 13:57:47.411775 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411676    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:47.411775 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.411745    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:47.411938 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411816    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:57:47.411938 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411870    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 13:57:47.411938 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411887    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 13:57:47.411938 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.411873    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.412163 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.412144    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qj52f\""
Sep 11 13:57:47.413950 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.413927    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.415430 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.415344    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:52:46 +0000 UTC" deadline="2028-03-01 07:26:05.624293043 +0000 UTC"
Sep 11 13:57:47.415430 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.415374    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12881h28m18.208923783s"
Sep 11 13:57:47.417601 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.416822    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.417601 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.416887    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.417601 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.417487    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.419388 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.418601    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kwkxt\""
Sep 11 13:57:47.419388 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.419067    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 13:57:47.419536 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.419421    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.419717 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.419694    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 13:57:47.419818 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.419797    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.420163 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.420110    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:47.420588 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.420341    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d7p27\""
Sep 11 13:57:47.420588 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.420433    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 13:57:47.422296 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.422265    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.422515 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.422496    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 13:57:47.422621 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.422567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 13:57:47.423288 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.423269    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.425871 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.425852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.426062 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.426031    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s7ltf\""
Sep 11 13:57:47.426166 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.426119    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 13:57:47.426315 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.426299    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.426406 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.426386    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.428376 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.428357    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.428489 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.428404    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.428489 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.428415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skslg\""
Sep 11 13:57:47.428489 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.428478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.430842 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.430822    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qn4z6\""
Sep 11 13:57:47.430992 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.430968    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 13:57:47.431123 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.431086    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 13:57:47.431123 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.431114    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.431271 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.431122    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 13:57:47.434575 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.434554    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 13:57:47.434771 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.434756    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mr5fm\""
Sep 11 13:57:47.437634 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.437614    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c57b5"
Sep 11 13:57:47.446967 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.446945    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c57b5"
Sep 11 13:57:47.500523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.500506    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 13:57:47.505481 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.505581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-run\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.505581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505526    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-host\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.505581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-kubelet\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.505581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505561    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85phb\" (UniqueName: \"kubernetes.io/projected/dd7aa459-b550-4414-b698-38ff47a50c8a-kube-api-access-85phb\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.505790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdvf\" (UniqueName: \"kubernetes.io/projected/16b5dea4-483a-4691-a62a-11a6db607f59-kube-api-access-5sdvf\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.505790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysconfig\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.505790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b5dea4-483a-4691-a62a-11a6db607f59-tmp\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.505790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-cnibin\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.505790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-socket-dir-parent\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.505790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505785    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-netns\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6922043-21b1-4117-90b6-8a6b54f09902-ovn-node-metrics-cert\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-socket-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-sys\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.505998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-cni-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f293fae-8105-42db-a858-0e1489da0d64-multus-daemon-config\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.506119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-sys-fs\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-systemd\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f293fae-8105-42db-a858-0e1489da0d64-cni-binary-copy\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506194    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-cnibin\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16b5dea4-483a-4691-a62a-11a6db607f59-etc-tuned\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506239    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-hosts-file\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-systemd-units\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-slash\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-env-overrides\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/902b2429-8e75-40e2-b3fd-b7e1a72ee32b-agent-certs\") pod \"konnectivity-agent-62pnq\" (UID: \"902b2429-8e75-40e2-b3fd-b7e1a72ee32b\") " pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-hostroot\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvxl\" (UniqueName: \"kubernetes.io/projected/8f293fae-8105-42db-a858-0e1489da0d64-kube-api-access-nvvxl\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-kubelet\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.506462 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4qg\" (UniqueName: \"kubernetes.io/projected/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-kube-api-access-2v4qg\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506518    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysctl-conf\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-k8s-cni-cncf-io\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-ovnkube-config\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-lib-modules\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-system-cni-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd7aa459-b550-4414-b698-38ff47a50c8a-host-slash\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506738    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxrb\" (UniqueName: \"kubernetes.io/projected/a6922043-21b1-4117-90b6-8a6b54f09902-kube-api-access-tdxrb\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506813    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-conf-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-log-socket\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2gl\" (UniqueName: \"kubernetes.io/projected/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-kube-api-access-ss2gl\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznpg\" (UniqueName: \"kubernetes.io/projected/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-kube-api-access-qznpg\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8rk\" (UniqueName: \"kubernetes.io/projected/dd187b1d-bbab-45a2-88a3-1d32dec00e36-kube-api-access-hj8rk\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507097 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.506980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-cni-netd\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysctl-d\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-etc-kubernetes\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-systemd\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-ovn\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/902b2429-8e75-40e2-b3fd-b7e1a72ee32b-konnectivity-ca\") pod \"konnectivity-agent-62pnq\" (UID: \"902b2429-8e75-40e2-b3fd-b7e1a72ee32b\") " pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd7aa459-b550-4414-b698-38ff47a50c8a-iptables-alerter-script\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-var-lib-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04cb311f-1d53-4152-9f5d-66fb2e50ca01-kubelet-config\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-system-cni-dir\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507384    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507411    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-multus-certs\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-registration-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-kubernetes\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.507661 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-var-lib-kubelet\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-host\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04cb311f-1d53-4152-9f5d-66fb2e50ca01-dbus\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-cni-multus\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-serviceca\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-modprobe-d\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-cni-bin\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-node-log\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-cni-bin\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-device-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-tmp-dir\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507893    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-ovnkube-script-lib\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.507965    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.508009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbd77\" (UniqueName: \"kubernetes.io/projected/b798c22e-4497-47e4-826a-a54445aef97f-kube-api-access-bbd77\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.508247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.508056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-run-netns\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.508769 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.508081    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-etc-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.508769 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.508105    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.537133 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.537094    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6841457108083885ee66eb5ffa556d37.slice/crio-0583555a1b3feae2b1c3656c1127f90622dff5911a417c52b7af27acd787d238 WatchSource:0}: Error finding container 0583555a1b3feae2b1c3656c1127f90622dff5911a417c52b7af27acd787d238: Status 404 returned error can't find the container with id 0583555a1b3feae2b1c3656c1127f90622dff5911a417c52b7af27acd787d238
Sep 11 13:57:47.537348 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.537331    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea5c1f950c94bad71aa5473962c9e7.slice/crio-4b2d87a1150cd921e7c3960c870457002cc3040c01e78edeaaded3318b32d382 WatchSource:0}: Error finding container 4b2d87a1150cd921e7c3960c870457002cc3040c01e78edeaaded3318b32d382: Status 404 returned error can't find the container with id 4b2d87a1150cd921e7c3960c870457002cc3040c01e78edeaaded3318b32d382
Sep 11 13:57:47.543795 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.542482    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:57:47.609221 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16b5dea4-483a-4691-a62a-11a6db607f59-etc-tuned\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.609221 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-hosts-file\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-systemd-units\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-slash\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-env-overrides\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/902b2429-8e75-40e2-b3fd-b7e1a72ee32b-agent-certs\") pod \"konnectivity-agent-62pnq\" (UID: \"902b2429-8e75-40e2-b3fd-b7e1a72ee32b\") " pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-hostroot\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609322    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-hosts-file\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-systemd-units\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609339    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-slash\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609328    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvxl\" (UniqueName: \"kubernetes.io/projected/8f293fae-8105-42db-a858-0e1489da0d64-kube-api-access-nvvxl\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609375    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-hostroot\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-kubelet\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4qg\" (UniqueName: \"kubernetes.io/projected/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-kube-api-access-2v4qg\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysctl-conf\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-k8s-cni-cncf-io\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-ovnkube-config\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609553    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-lib-modules\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-kubelet\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-system-cni-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd7aa459-b550-4414-b698-38ff47a50c8a-host-slash\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-k8s-cni-cncf-io\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxrb\" (UniqueName: \"kubernetes.io/projected/a6922043-21b1-4117-90b6-8a6b54f09902-kube-api-access-tdxrb\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysctl-conf\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.609682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-lib-modules\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-conf-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-system-cni-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609704    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd7aa459-b550-4414-b698-38ff47a50c8a-host-slash\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-log-socket\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609751    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.609782    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2gl\" (UniqueName: \"kubernetes.io/projected/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-kube-api-access-ss2gl\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-log-socket\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-conf-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.609895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznpg\" (UniqueName: \"kubernetes.io/projected/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-kube-api-access-qznpg\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.609966    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:48.109946432 +0000 UTC m=+2.073561412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8rk\" (UniqueName: \"kubernetes.io/projected/dd187b1d-bbab-45a2-88a3-1d32dec00e36-kube-api-access-hj8rk\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-cni-netd\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610232    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-env-overrides\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.610574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysctl-d\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-ovnkube-config\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-etc-kubernetes\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-cni-netd\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-systemd\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-ovn\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-etc-kubernetes\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610445    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysctl-d\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610453    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-systemd\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/902b2429-8e75-40e2-b3fd-b7e1a72ee32b-konnectivity-ca\") pod \"konnectivity-agent-62pnq\" (UID: \"902b2429-8e75-40e2-b3fd-b7e1a72ee32b\") " pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-ovn\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd7aa459-b550-4414-b698-38ff47a50c8a-iptables-alerter-script\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610534    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-var-lib-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04cb311f-1d53-4152-9f5d-66fb2e50ca01-kubelet-config\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.610545    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-system-cni-dir\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.611590 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-var-lib-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.610624    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:48.110605754 +0000 UTC m=+2.074220711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04cb311f-1d53-4152-9f5d-66fb2e50ca01-kubelet-config\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610692    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-system-cni-dir\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-multus-certs\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-multus-certs\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-run-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-registration-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-kubernetes\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-var-lib-kubelet\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-host\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610879    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-registration-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610886    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04cb311f-1d53-4152-9f5d-66fb2e50ca01-dbus\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-kubernetes\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-cni-multus\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04cb311f-1d53-4152-9f5d-66fb2e50ca01-dbus\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:47.612574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-serviceca\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610959    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-var-lib-kubelet\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-modprobe-d\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-host\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.610995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-cni-bin\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-cni-multus\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-node-log\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611060    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-cni-bin\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-cni-bin\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611085    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-modprobe-d\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-node-log\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611123    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-device-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611176    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-tmp-dir\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-ovnkube-script-lib\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-device-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.613523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/902b2429-8e75-40e2-b3fd-b7e1a72ee32b-konnectivity-ca\") pod \"konnectivity-agent-62pnq\" (UID: \"902b2429-8e75-40e2-b3fd-b7e1a72ee32b\") " pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbd77\" (UniqueName: \"kubernetes.io/projected/b798c22e-4497-47e4-826a-a54445aef97f-kube-api-access-bbd77\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-cni-bin\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd7aa459-b550-4414-b698-38ff47a50c8a-iptables-alerter-script\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-serviceca\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611592    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-run-netns\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-host-run-netns\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-tmp-dir\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-etc-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611697    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611728    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6922043-21b1-4117-90b6-8a6b54f09902-etc-openvswitch\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-run\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-host\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-kubelet\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85phb\" (UniqueName: \"kubernetes.io/projected/dd7aa459-b550-4414-b698-38ff47a50c8a-kube-api-access-85phb\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-host\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdvf\" (UniqueName: \"kubernetes.io/projected/16b5dea4-483a-4691-a62a-11a6db607f59-kube-api-access-5sdvf\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-var-lib-kubelet\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-run\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysconfig\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b5dea4-483a-4691-a62a-11a6db607f59-tmp\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-sysconfig\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-cnibin\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-socket-dir-parent\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.611992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-netns\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6922043-21b1-4117-90b6-8a6b54f09902-ovnkube-script-lib\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612058    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6922043-21b1-4117-90b6-8a6b54f09902-ovn-node-metrics-cert\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-socket-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-sys\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.614976 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-cni-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f293fae-8105-42db-a858-0e1489da0d64-multus-daemon-config\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-sys-fs\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-systemd\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f293fae-8105-42db-a858-0e1489da0d64-cni-binary-copy\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-cnibin\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612294    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-sys\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-cni-dir\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-multus-socket-dir-parent\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16b5dea4-483a-4691-a62a-11a6db607f59-etc-systemd\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-host-run-netns\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f293fae-8105-42db-a858-0e1489da0d64-multus-daemon-config\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-sys-fs\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f293fae-8105-42db-a858-0e1489da0d64-cni-binary-copy\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612945    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-socket-dir\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f293fae-8105-42db-a858-0e1489da0d64-cnibin\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.612995    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b798c22e-4497-47e4-826a-a54445aef97f-cnibin\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.615567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.613087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.616115 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.613318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b798c22e-4497-47e4-826a-a54445aef97f-cni-binary-copy\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.616115 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.613682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16b5dea4-483a-4691-a62a-11a6db607f59-etc-tuned\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.616115 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.613841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/902b2429-8e75-40e2-b3fd-b7e1a72ee32b-agent-certs\") pod \"konnectivity-agent-62pnq\" (UID: \"902b2429-8e75-40e2-b3fd-b7e1a72ee32b\") " pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.616115 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.614250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b5dea4-483a-4691-a62a-11a6db607f59-tmp\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.616115 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.614659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6922043-21b1-4117-90b6-8a6b54f09902-ovn-node-metrics-cert\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.620125 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.620051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4qg\" (UniqueName: \"kubernetes.io/projected/d04e7f93-5071-4cd8-a46e-f8c1d96799a1-kube-api-access-2v4qg\") pod \"aws-ebs-csi-driver-node-rrxl5\" (UID: \"d04e7f93-5071-4cd8-a46e-f8c1d96799a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.620352 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.620331    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:57:47.620438 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.620363    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:57:47.620438 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.620379    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:47.620438 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:47.620433    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:48.120416264 +0000 UTC m=+2.084031235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:47.620608 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.620551    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2gl\" (UniqueName: \"kubernetes.io/projected/d8eb0a86-1eea-4d55-87a7-1bca0e1ed936-kube-api-access-ss2gl\") pod \"node-ca-6wjfc\" (UID: \"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936\") " pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.621344 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.621276    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvxl\" (UniqueName: \"kubernetes.io/projected/8f293fae-8105-42db-a858-0e1489da0d64-kube-api-access-nvvxl\") pod \"multus-7hqrb\" (UID: \"8f293fae-8105-42db-a858-0e1489da0d64\") " pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.621430 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.621352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8rk\" (UniqueName: \"kubernetes.io/projected/dd187b1d-bbab-45a2-88a3-1d32dec00e36-kube-api-access-hj8rk\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:47.621530 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.621484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznpg\" (UniqueName: \"kubernetes.io/projected/c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2-kube-api-access-qznpg\") pod \"node-resolver-6mxh4\" (UID: \"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2\") " pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.622361 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.622337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85phb\" (UniqueName: \"kubernetes.io/projected/dd7aa459-b550-4414-b698-38ff47a50c8a-kube-api-access-85phb\") pod \"iptables-alerter-2tnsf\" (UID: \"dd7aa459-b550-4414-b698-38ff47a50c8a\") " pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.622691 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.622668    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxrb\" (UniqueName: \"kubernetes.io/projected/a6922043-21b1-4117-90b6-8a6b54f09902-kube-api-access-tdxrb\") pod \"ovnkube-node-dsrq8\" (UID: \"a6922043-21b1-4117-90b6-8a6b54f09902\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.622864 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.622847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbd77\" (UniqueName: \"kubernetes.io/projected/b798c22e-4497-47e4-826a-a54445aef97f-kube-api-access-bbd77\") pod \"multus-additional-cni-plugins-sjdj6\" (UID: \"b798c22e-4497-47e4-826a-a54445aef97f\") " pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.623252 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.623236    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdvf\" (UniqueName: \"kubernetes.io/projected/16b5dea4-483a-4691-a62a-11a6db607f59-kube-api-access-5sdvf\") pod \"tuned-rl2qm\" (UID: \"16b5dea4-483a-4691-a62a-11a6db607f59\") " pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.730105 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.730084    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:57:47.740726 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.740706    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902b2429_8e75_40e2_b3fd_b7e1a72ee32b.slice/crio-b5bc13d20a3dac91ad402a31eefbd5c81a6702d102324c8e9dc71c97babc0b09 WatchSource:0}: Error finding container b5bc13d20a3dac91ad402a31eefbd5c81a6702d102324c8e9dc71c97babc0b09: Status 404 returned error can't find the container with id b5bc13d20a3dac91ad402a31eefbd5c81a6702d102324c8e9dc71c97babc0b09
Sep 11 13:57:47.747011 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.746991    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mxh4"
Sep 11 13:57:47.754690 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.754671    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sjdj6"
Sep 11 13:57:47.755652 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.755621    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3de9bfd_b0a4_4bc3_802b_d354f5a1a2f2.slice/crio-052810f80c8728b0edaaf59da623303dd351566d88f7b03d5a66e845b778854f WatchSource:0}: Error finding container 052810f80c8728b0edaaf59da623303dd351566d88f7b03d5a66e845b778854f: Status 404 returned error can't find the container with id 052810f80c8728b0edaaf59da623303dd351566d88f7b03d5a66e845b778854f
Sep 11 13:57:47.762969 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.762939    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb798c22e_4497_47e4_826a_a54445aef97f.slice/crio-2b53c599bf6ec098e9a80e7b738c6ae92d2d379897b773da2427fd8b6d632b29 WatchSource:0}: Error finding container 2b53c599bf6ec098e9a80e7b738c6ae92d2d379897b773da2427fd8b6d632b29: Status 404 returned error can't find the container with id 2b53c599bf6ec098e9a80e7b738c6ae92d2d379897b773da2427fd8b6d632b29
Sep 11 13:57:47.778647 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.778626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2tnsf"
Sep 11 13:57:47.784941 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.784923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:57:47.786543 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.786523    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:57:47.786921 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.786902    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7aa459_b550_4414_b698_38ff47a50c8a.slice/crio-48a221a0944a9456a0a4bb4ab28ae638417b0de075d45a25c4e477d2d02a4b73 WatchSource:0}: Error finding container 48a221a0944a9456a0a4bb4ab28ae638417b0de075d45a25c4e477d2d02a4b73: Status 404 returned error can't find the container with id 48a221a0944a9456a0a4bb4ab28ae638417b0de075d45a25c4e477d2d02a4b73
Sep 11 13:57:47.794856 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.794838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5"
Sep 11 13:57:47.794993 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.794975    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6922043_21b1_4117_90b6_8a6b54f09902.slice/crio-1aa983842eacc7104b24f9609d824a7a5837cd2cc8e4b204fc59fb0beb057d20 WatchSource:0}: Error finding container 1aa983842eacc7104b24f9609d824a7a5837cd2cc8e4b204fc59fb0beb057d20: Status 404 returned error can't find the container with id 1aa983842eacc7104b24f9609d824a7a5837cd2cc8e4b204fc59fb0beb057d20
Sep 11 13:57:47.802700 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.802679    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04e7f93_5071_4cd8_a46e_f8c1d96799a1.slice/crio-0808770846b7bc718a1274360d47a49a362dd2905edf82bc053cd43a09a30937 WatchSource:0}: Error finding container 0808770846b7bc718a1274360d47a49a362dd2905edf82bc053cd43a09a30937: Status 404 returned error can't find the container with id 0808770846b7bc718a1274360d47a49a362dd2905edf82bc053cd43a09a30937
Sep 11 13:57:47.815382 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.815362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rl2qm"
Sep 11 13:57:47.823129 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.823088    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b5dea4_483a_4691_a62a_11a6db607f59.slice/crio-3efb2abde5efe5c755edffd48f7d9ab31ab637301f5c9fe6af4e350dec364b4b WatchSource:0}: Error finding container 3efb2abde5efe5c755edffd48f7d9ab31ab637301f5c9fe6af4e350dec364b4b: Status 404 returned error can't find the container with id 3efb2abde5efe5c755edffd48f7d9ab31ab637301f5c9fe6af4e350dec364b4b
Sep 11 13:57:47.846060 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.846024    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wjfc"
Sep 11 13:57:47.850263 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.850245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hqrb"
Sep 11 13:57:47.854013 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.853992    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eb0a86_1eea_4d55_87a7_1bca0e1ed936.slice/crio-c42521ecedbccb5901ae77def6c1bc7daedfee1be12dc308a072fc9130b53871 WatchSource:0}: Error finding container c42521ecedbccb5901ae77def6c1bc7daedfee1be12dc308a072fc9130b53871: Status 404 returned error can't find the container with id c42521ecedbccb5901ae77def6c1bc7daedfee1be12dc308a072fc9130b53871
Sep 11 13:57:47.858278 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:57:47.858258    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f293fae_8105_42db_a858_0e1489da0d64.slice/crio-8213aec525c76af5fd5b5c2d3693a1ad385b19b3fa18c1fb31f9f70198bba62a WatchSource:0}: Error finding container 8213aec525c76af5fd5b5c2d3693a1ad385b19b3fa18c1fb31f9f70198bba62a: Status 404 returned error can't find the container with id 8213aec525c76af5fd5b5c2d3693a1ad385b19b3fa18c1fb31f9f70198bba62a
Sep 11 13:57:47.918477 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:47.918460    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:57:48.116042 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.115958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:48.116042 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.116015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:48.116295 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.116158    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:48.116295 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.116220    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:49.116200004 +0000 UTC m=+3.079814964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:48.116642 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.116618    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:48.116736 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.116678    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:49.116661247 +0000 UTC m=+3.080276210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:48.216716 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.216684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:48.216876 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.216825    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:57:48.216876 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.216844    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:57:48.216876 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.216856    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:48.217047 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:48.216909    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:49.216890745 +0000 UTC m=+3.180505705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:48.450247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.450183    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:52:47 +0000 UTC" deadline="2028-05-10 08:41:37.661062968 +0000 UTC"
Sep 11 13:57:48.450247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.450247    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14562h43m49.210820226s"
Sep 11 13:57:48.513873 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.513787    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hqrb" event={"ID":"8f293fae-8105-42db-a858-0e1489da0d64","Type":"ContainerStarted","Data":"8213aec525c76af5fd5b5c2d3693a1ad385b19b3fa18c1fb31f9f70198bba62a"}
Sep 11 13:57:48.523324 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.523285    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rl2qm" event={"ID":"16b5dea4-483a-4691-a62a-11a6db607f59","Type":"ContainerStarted","Data":"3efb2abde5efe5c755edffd48f7d9ab31ab637301f5c9fe6af4e350dec364b4b"}
Sep 11 13:57:48.530639 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.530549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5" event={"ID":"d04e7f93-5071-4cd8-a46e-f8c1d96799a1","Type":"ContainerStarted","Data":"0808770846b7bc718a1274360d47a49a362dd2905edf82bc053cd43a09a30937"}
Sep 11 13:57:48.547012 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.546980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2tnsf" event={"ID":"dd7aa459-b550-4414-b698-38ff47a50c8a","Type":"ContainerStarted","Data":"48a221a0944a9456a0a4bb4ab28ae638417b0de075d45a25c4e477d2d02a4b73"}
Sep 11 13:57:48.564251 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.564213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mxh4" event={"ID":"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2","Type":"ContainerStarted","Data":"052810f80c8728b0edaaf59da623303dd351566d88f7b03d5a66e845b778854f"}
Sep 11 13:57:48.577221 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.577193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-62pnq" event={"ID":"902b2429-8e75-40e2-b3fd-b7e1a72ee32b","Type":"ContainerStarted","Data":"b5bc13d20a3dac91ad402a31eefbd5c81a6702d102324c8e9dc71c97babc0b09"}
Sep 11 13:57:48.582067 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.581837    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wjfc" event={"ID":"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936","Type":"ContainerStarted","Data":"c42521ecedbccb5901ae77def6c1bc7daedfee1be12dc308a072fc9130b53871"}
Sep 11 13:57:48.591123 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.591096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"1aa983842eacc7104b24f9609d824a7a5837cd2cc8e4b204fc59fb0beb057d20"}
Sep 11 13:57:48.600647 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.600621    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerStarted","Data":"2b53c599bf6ec098e9a80e7b738c6ae92d2d379897b773da2427fd8b6d632b29"}
Sep 11 13:57:48.607793 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.607766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal" event={"ID":"49ea5c1f950c94bad71aa5473962c9e7","Type":"ContainerStarted","Data":"4b2d87a1150cd921e7c3960c870457002cc3040c01e78edeaaded3318b32d382"}
Sep 11 13:57:48.617393 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.617368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal" event={"ID":"6841457108083885ee66eb5ffa556d37","Type":"ContainerStarted","Data":"0583555a1b3feae2b1c3656c1127f90622dff5911a417c52b7af27acd787d238"}
Sep 11 13:57:48.817371 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:48.817291    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:57:49.124889 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.124581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:49.124889 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.124876    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:49.125226 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.124761    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:49.125226 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.124987    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:51.124965529 +0000 UTC m=+5.088580485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:49.125226 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.125037    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:49.125226 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.125108    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:51.125074535 +0000 UTC m=+5.088689487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:49.225350 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.225314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:49.225532 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.225479    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:57:49.225532 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.225500    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:57:49.225532 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.225514    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:49.225731 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.225569    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:51.225550771 +0000 UTC m=+5.189165740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:49.451468 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.451422    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:52:47 +0000 UTC" deadline="2028-04-19 15:22:56.682199636 +0000 UTC"
Sep 11 13:57:49.451943 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.451561    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14065h25m7.230645496s"
Sep 11 13:57:49.498510 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.497686    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:49.498510 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.497816    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:49.498510 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.498232    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:49.498510 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.498324    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:57:49.498510 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:49.498386    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:49.498510 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:49.498473    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:50.211806 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:50.211773    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:57:51.141488 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:51.141446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:51.141947 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:51.141512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:51.141947 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.141692    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:51.141947 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.141759    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:55.141738406 +0000 UTC m=+9.105353389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:51.142231 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.142210    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:51.142310 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.142273    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:55.142256828 +0000 UTC m=+9.105871784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:51.243047 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:51.242948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:51.243220 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.243129    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:57:51.243220 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.243152    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:57:51.243220 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.243165    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:51.243387 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.243232    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:55.243212348 +0000 UTC m=+9.206827305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:51.497727 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:51.497653    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:51.497882 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.497760    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:51.498132 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:51.498109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:51.498233 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.498211    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:57:51.498304 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:51.498247    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:51.498363 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:51.498308    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:53.497739 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:53.497710    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:53.498237 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:53.497754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:53.498237 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:53.497718    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:53.498237 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:53.497841    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:57:53.498237 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:53.497927    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:53.498237 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:53.498028    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:55.173356 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:55.173329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:55.173758 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:55.173365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:55.173758 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.173475    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:55.173758 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.173525    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:03.17351115 +0000 UTC m=+17.137126106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:57:55.173758 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.173475    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:55.173758 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.173598    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:03.173586318 +0000 UTC m=+17.137201279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:57:55.274016 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:55.273975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:55.274617 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.274216    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:57:55.274617 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.274240    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:57:55.274617 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.274253    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:55.274617 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.274311    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:03.274293806 +0000 UTC m=+17.237908772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:57:55.497547 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:55.497503    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:55.497547 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:55.497530    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:55.497777 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:55.497627    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:55.497777 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.497630    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:55.497777 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.497727    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:57:55.497936 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:55.497812    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:57.498243 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:57.498207    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:57.498243 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:57.498234    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:57.498712 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:57.498207    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:57.498712 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:57.498323    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:57:57.498712 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:57.498459    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:57.498712 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:57.498541    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:59.497745 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:59.497680    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:57:59.498286 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:59.497790    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:57:59.498286 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:57:59.497846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:57:59.498286 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:59.497807    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:57:59.498286 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:59.497949    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:57:59.498286 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:57:59.498040    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:01.498159 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:01.498128    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:01.498572 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:01.498226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:01.498572 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:01.498233    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:01.498572 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:01.498254    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:01.498572 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:01.498339    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:01.498572 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:01.498424    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:03.230392 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:03.230354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:03.230840 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:03.230404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:03.230840 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.230497    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:58:03.230840 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.230513    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:58:03.230840 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.230563    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:19.230547508 +0000 UTC m=+33.194162464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:58:03.230840 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.230578    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:19.230571445 +0000 UTC m=+33.194186396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:58:03.331398 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:03.331371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:03.331586 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.331566    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:58:03.331636 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.331595    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:58:03.331636 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.331609    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:58:03.331705 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.331670    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:19.331653517 +0000 UTC m=+33.295268475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:58:03.498251 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:03.498174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:03.498251 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:03.498193    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:03.498449 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:03.498276    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:03.498449 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.498287    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:03.498449 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.498374    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:03.498449 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:03.498426    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:05.497913 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:05.497884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:05.498368 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:05.497896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:05.498368 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:05.498008    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:05.498368 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:05.497896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:05.498368 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:05.498108    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:05.498368 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:05.498193    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:07.498370 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.498208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:07.498828 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.498234    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:07.498828 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:07.498451    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:07.498828 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.498267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:07.498828 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:07.498533    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:07.498828 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:07.498594    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:07.668938 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.668889    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 13:58:07.669274 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669255    2558 generic.go:356] "Generic (PLEG): container finished" podID="a6922043-21b1-4117-90b6-8a6b54f09902" containerID="e0983dc63d3ca87d75e84d052e390c6693b0582cb6b6b9f169df74965639a869" exitCode=1
Sep 11 13:58:07.669332 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"2bef1962b1e3ccb8fac9ae46291be807b65256d12e103cca9499ad5c7b69f91d"}
Sep 11 13:58:07.669381 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"3385ac39ad1646103ad61333b9cf5949fd2804ed45ba292db222b19874001e36"}
Sep 11 13:58:07.669433 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"15679da915e5a48d594fb188d35456dd80544453d202959dc68d295bc9042d74"}
Sep 11 13:58:07.669474 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"c36bfb7288bae1499d02f37c123aa62febb8d750bc9dc97e3821e673b778f1b8"}
Sep 11 13:58:07.669507 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669489    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerDied","Data":"e0983dc63d3ca87d75e84d052e390c6693b0582cb6b6b9f169df74965639a869"}
Sep 11 13:58:07.669539 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.669519    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"5a2d146865ac1c15778409aa55cc3e704d7311dfbf590d455e18983ec2aff943"}
Sep 11 13:58:07.670765 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.670743    2558 generic.go:356] "Generic (PLEG): container finished" podID="b798c22e-4497-47e4-826a-a54445aef97f" containerID="bfb3b250c3afcd91e12ca5041e9d08e7026e095dd55b0b515451559d1419d569" exitCode=0
Sep 11 13:58:07.670833 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.670814    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerDied","Data":"bfb3b250c3afcd91e12ca5041e9d08e7026e095dd55b0b515451559d1419d569"}
Sep 11 13:58:07.672401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.672378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal" event={"ID":"49ea5c1f950c94bad71aa5473962c9e7","Type":"ContainerStarted","Data":"1fb70b86043a1fb3d962293d6ae07627c0f8042592b9bd27f86fa3935d2d26c2"}
Sep 11 13:58:07.673955 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.673935    2558 generic.go:356] "Generic (PLEG): container finished" podID="6841457108083885ee66eb5ffa556d37" containerID="d7ffef7a8921d75ecd31fa61c843b1c71677b21a51fd1c485350d07666bb043d" exitCode=0
Sep 11 13:58:07.674066 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.674009    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal" event={"ID":"6841457108083885ee66eb5ffa556d37","Type":"ContainerDied","Data":"d7ffef7a8921d75ecd31fa61c843b1c71677b21a51fd1c485350d07666bb043d"}
Sep 11 13:58:07.675448 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.675428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hqrb" event={"ID":"8f293fae-8105-42db-a858-0e1489da0d64","Type":"ContainerStarted","Data":"00893536bc78e4f2df0f03e8267038ac7b59e74bed8ee87ff513114829f5867a"}
Sep 11 13:58:07.677081 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.677059    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rl2qm" event={"ID":"16b5dea4-483a-4691-a62a-11a6db607f59","Type":"ContainerStarted","Data":"5551aaaffe00488c099d30349a6443f2d4e096e4f73a1ebf411f45c1352eb739"}
Sep 11 13:58:07.678574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.678548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5" event={"ID":"d04e7f93-5071-4cd8-a46e-f8c1d96799a1","Type":"ContainerStarted","Data":"1941e0a906cb282fa66d5a2e5ca9ef2ea07230bcc5f6ac5576b8168c3aa18479"}
Sep 11 13:58:07.679862 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.679844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mxh4" event={"ID":"c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2","Type":"ContainerStarted","Data":"c5cf3d7833dd152c07f50f26aabfc9a3d5ea667bca55dd5084494f9c46151ba2"}
Sep 11 13:58:07.681145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.681120    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-62pnq" event={"ID":"902b2429-8e75-40e2-b3fd-b7e1a72ee32b","Type":"ContainerStarted","Data":"0a1184cab9a868d25136674ad12c9d78c7ebadaa97e048835e3ad60baebf09a2"}
Sep 11 13:58:07.682673 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.682654    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wjfc" event={"ID":"d8eb0a86-1eea-4d55-87a7-1bca0e1ed936","Type":"ContainerStarted","Data":"c7aa0881671857ffe12504a6a9a87366cbcada47b78404827c5876018d0ac10a"}
Sep 11 13:58:07.708911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.708871    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-62pnq" podStartSLOduration=1.635205628 podStartE2EDuration="20.708859713s" totalImagesPullingTime="19.073654085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:07.708640918 +0000 UTC m=+21.672255893" watchObservedRunningTime="2026-09-11 13:58:07.708859713 +0000 UTC m=+21.672474713"
Sep 11 13:58:07.726480 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.726449    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6wjfc" podStartSLOduration=1.765988372 podStartE2EDuration="20.726438915s" totalImagesPullingTime="18.960450543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:07.726090719 +0000 UTC m=+21.689705692" watchObservedRunningTime="2026-09-11 13:58:07.726438915 +0000 UTC m=+21.690053889"
Sep 11 13:58:07.743672 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.743642    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6mxh4" podStartSLOduration=1.6846111929999998 podStartE2EDuration="20.743632187s" totalImagesPullingTime="19.059020994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:07.743137947 +0000 UTC m=+21.706752922" watchObservedRunningTime="2026-09-11 13:58:07.743632187 +0000 UTC m=+21.707247161"
Sep 11 13:58:07.768455 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.768425    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rl2qm" podStartSLOduration=1.77730094 podStartE2EDuration="20.768415067s" totalImagesPullingTime="18.991114127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:07.768084094 +0000 UTC m=+21.731699069" watchObservedRunningTime="2026-09-11 13:58:07.768415067 +0000 UTC m=+21.732030037"
Sep 11 13:58:07.790597 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.790560    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7hqrb" podStartSLOduration=1.676279761 podStartE2EDuration="20.790550038s" totalImagesPullingTime="19.114270277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:07.790493024 +0000 UTC m=+21.754108001" watchObservedRunningTime="2026-09-11 13:58:07.790550038 +0000 UTC m=+21.754165013"
Sep 11 13:58:07.806413 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.806382    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-144.ec2.internal" podStartSLOduration=20.806372175 podStartE2EDuration="20.806372175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:07.806115772 +0000 UTC m=+21.769730745" watchObservedRunningTime="2026-09-11 13:58:07.806372175 +0000 UTC m=+21.769987148"
Sep 11 13:58:07.835567 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.835542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:07.911266 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:07.911244    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:08.016002 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.015955    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:08.667540 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.667494    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:08.686927 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.686813    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2tnsf" event={"ID":"dd7aa459-b550-4414-b698-38ff47a50c8a","Type":"ContainerStarted","Data":"c6aea20e77782427185100205b06d74c613c00765ab327fd70f989f76b8fed81"}
Sep 11 13:58:08.703587 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.703532    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2tnsf" podStartSLOduration=2.678054577 podStartE2EDuration="21.703518213s" totalImagesPullingTime="19.025463636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:08.70322904 +0000 UTC m=+22.666844015" watchObservedRunningTime="2026-09-11 13:58:08.703518213 +0000 UTC m=+22.667133186"
Sep 11 13:58:08.733066 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.733038    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:08.884731 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.884702    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:08.967152 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:08.966891    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 13:58:09.003431 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.003324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:09.080229 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.080189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:09.283537 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.283499    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:09.356428 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.356311    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:09.458936 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.458832    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:58:08.966910783Z","UUID":"232d4391-f5cf-4c6f-bd0b-1ad6a546e948","Handler":null,"Name":"","Endpoint":""}
Sep 11 13:58:09.460847 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.460403    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 13:58:09.460847 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.460437    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 13:58:09.497461 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.497439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:09.497592 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:09.497552    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:09.497592 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.497585    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:09.497721 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:09.497696    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:09.497857 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.497751    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:09.497857 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:09.497829    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:09.691382 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.691345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal" event={"ID":"6841457108083885ee66eb5ffa556d37","Type":"ContainerStarted","Data":"92560c87d5d979312072ff7c23d00db77330d7b200bc389dbc231d8015e73958"}
Sep 11 13:58:09.693584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.693548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5" event={"ID":"d04e7f93-5071-4cd8-a46e-f8c1d96799a1","Type":"ContainerStarted","Data":"e89434a633eeafb4c987b99b724e0722348ab7a0354f5613cc4f7aeb464c1fe4"}
Sep 11 13:58:09.707912 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:09.707864    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-144.ec2.internal" podStartSLOduration=22.707848446 podStartE2EDuration="22.707848446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:09.707637199 +0000 UTC m=+23.671252203" watchObservedRunningTime="2026-09-11 13:58:09.707848446 +0000 UTC m=+23.671463420"
Sep 11 13:58:10.459169 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.458935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:58:10.459506 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.459487    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:58:10.698322 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.698286    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5" event={"ID":"d04e7f93-5071-4cd8-a46e-f8c1d96799a1","Type":"ContainerStarted","Data":"bdde5ebfe9026439b793638df3c5b2557d5c8d24a2b817ac2afec70ff191fbf5"}
Sep 11 13:58:10.701988 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.701961    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 13:58:10.702494 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.702460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"6d8b86b8ffe3b7f8d20eab7c4777814df411db58e4db741829b7dd390c61626c"}
Sep 11 13:58:10.702625 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.702554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:58:10.703290 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.703271    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-62pnq"
Sep 11 13:58:10.719528 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.719447    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rrxl5" podStartSLOduration=2.218937938 podStartE2EDuration="23.719430857s" totalImagesPullingTime="21.500492919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:10.719089249 +0000 UTC m=+24.682704226" watchObservedRunningTime="2026-09-11 13:58:10.719430857 +0000 UTC m=+24.683045835"
Sep 11 13:58:10.792960 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:10.792931    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:11.497720 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:11.497692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:11.497896 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:11.497783    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:11.497896 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:11.497803    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:11.497896 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:11.497800    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:11.498017 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:11.497895    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:11.498017 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:11.497967    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:11.717428 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:11.717398    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:11.818243 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:11.818217    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:12.709871 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.709846    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 13:58:12.710323 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.710295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"a19d64b1ee6a5ce97d1601c7016d841ccc6e83f43bbe14e733fd7069cda1d0af"}
Sep 11 13:58:12.710554 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.710530    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:58:12.710684 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.710598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:58:12.710743 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.710727    2558 scope.go:117] "RemoveContainer" containerID="e0983dc63d3ca87d75e84d052e390c6693b0582cb6b6b9f169df74965639a869"
Sep 11 13:58:12.712431 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.712404    2558 generic.go:356] "Generic (PLEG): container finished" podID="b798c22e-4497-47e4-826a-a54445aef97f" containerID="663c900ee09d5d0f3b56c9a0a29550eaf31cfd212690dece196ebcd14ba40e73" exitCode=0
Sep 11 13:58:12.712535 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.712438    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerDied","Data":"663c900ee09d5d0f3b56c9a0a29550eaf31cfd212690dece196ebcd14ba40e73"}
Sep 11 13:58:12.728001 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:12.727977    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:58:13.114873 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.114633    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:13.497451 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.497413    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:13.497628 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.497456    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:13.497628 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.497453    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:13.497628 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:13.497528    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:13.497794 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:13.497634    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:13.497794 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:13.497722    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:13.716704 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.716679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerStarted","Data":"207f41a9869e9540d1b0b53778837e50fd55c0c79b35e5671f4992ef942a2978"}
Sep 11 13:58:13.719994 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.719976    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 13:58:13.720343 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.720320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" event={"ID":"a6922043-21b1-4117-90b6-8a6b54f09902","Type":"ContainerStarted","Data":"190b89324a6fe5ccee358b5ef49c724e32c3a5b263f790aa5d93730433982c01"}
Sep 11 13:58:13.720570 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.720554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:58:13.738561 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.738541    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:58:13.768990 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.768912    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8" podStartSLOduration=7.728873246 podStartE2EDuration="26.768895545s" totalImagesPullingTime="19.040022299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:13.767483704 +0000 UTC m=+27.731098677" watchObservedRunningTime="2026-09-11 13:58:13.768895545 +0000 UTC m=+27.732510523"
Sep 11 13:58:13.864601 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.864527    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bqdgz"]
Sep 11 13:58:13.864691 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.864660    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:13.864835 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:13.864812    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:13.865296 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.865274    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-th28r"]
Sep 11 13:58:13.865409 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.865365    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:13.865483 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:13.865464    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:13.865860 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.865836    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qtbqd"]
Sep 11 13:58:13.865953 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:13.865899    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:13.866019 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:13.865963    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:14.250719 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:14.250686    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:14.724507 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:14.724479    2558 generic.go:356] "Generic (PLEG): container finished" podID="b798c22e-4497-47e4-826a-a54445aef97f" containerID="207f41a9869e9540d1b0b53778837e50fd55c0c79b35e5671f4992ef942a2978" exitCode=0
Sep 11 13:58:14.724668 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:14.724541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerDied","Data":"207f41a9869e9540d1b0b53778837e50fd55c0c79b35e5671f4992ef942a2978"}
Sep 11 13:58:15.497573 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:15.497540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:15.497986 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:15.497580    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:15.497986 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:15.497602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:15.497986 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:15.497686    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:15.497986 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:15.497741    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:15.497986 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:15.497861    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:16.538660 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:16.538507    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:16.631099 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:16.631063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:16.732138 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:16.732114    2558 generic.go:356] "Generic (PLEG): container finished" podID="b798c22e-4497-47e4-826a-a54445aef97f" containerID="a057dbfce0b3ba5fc37b7abda5628ee0888f6b351817df184deb467640f2555c" exitCode=0
Sep 11 13:58:16.732247 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:16.732166    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerDied","Data":"a057dbfce0b3ba5fc37b7abda5628ee0888f6b351817df184deb467640f2555c"}
Sep 11 13:58:17.497593 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:17.497563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:17.497593 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:17.497583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:17.497844 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:17.497685    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:17.497844 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:17.497686    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:17.497844 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:17.497770    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:17.497961 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:17.497866    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:18.021879 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:18.021820    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:18.089609 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:18.089568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:19.251809 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:19.251761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:19.252291 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:19.251821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:19.252291 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.251915    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:58:19.252291 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.251997    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret podName:04cb311f-1d53-4152-9f5d-66fb2e50ca01 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.251977455 +0000 UTC m=+65.215592438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret") pod "global-pull-secret-syncer-qtbqd" (UID: "04cb311f-1d53-4152-9f5d-66fb2e50ca01") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:58:19.252291 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.251917    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:58:19.252291 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.252054    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.252042216 +0000 UTC m=+65.215657187 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:58:19.352787 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:19.352755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:19.352966 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.352937    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:58:19.352966 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.352959    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:58:19.353118 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.352973    2558 projected.go:196] Error preparing data for projected volume kube-api-access-x7s5z for pod openshift-network-diagnostics/network-check-target-bqdgz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:58:19.353118 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.353052    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z podName:f87e021d-43bf-40e2-b0bd-d251c9c19079 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.353014897 +0000 UTC m=+65.316629865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7s5z" (UniqueName: "kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z") pod "network-check-target-bqdgz" (UID: "f87e021d-43bf-40e2-b0bd-d251c9c19079") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:58:19.498198 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:19.498163    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:19.498409 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:19.498175    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:19.498409 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:19.498291    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:19.498409 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.498311    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th28r" podUID="dd187b1d-bbab-45a2-88a3-1d32dec00e36"
Sep 11 13:58:19.498568 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.498412    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qtbqd" podUID="04cb311f-1d53-4152-9f5d-66fb2e50ca01"
Sep 11 13:58:19.498568 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:19.498512    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqdgz" podUID="f87e021d-43bf-40e2-b0bd-d251c9c19079"
Sep 11 13:58:20.841779 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.841708    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-144.ec2.internal" event="NodeReady"
Sep 11 13:58:20.842149 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.841841    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 13:58:20.891743 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.891708    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94z9z"]
Sep 11 13:58:20.916182 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.916158    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mfdxs"]
Sep 11 13:58:20.916290 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.916264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:20.918705 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.918685    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tbm52\""
Sep 11 13:58:20.918814 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.918730    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 13:58:20.918814 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.918781    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 13:58:20.940812 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.940789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94z9z"]
Sep 11 13:58:20.940917 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.940826    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfdxs"]
Sep 11 13:58:20.940981 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.940914    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:20.943611 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.943590    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rh6t5\""
Sep 11 13:58:20.943729 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.943615    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 13:58:20.943729 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.943646    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 13:58:20.943729 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:20.943615    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 13:58:21.022638 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.022607    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:21.066005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.065978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c93581a-4009-4914-b149-b586d7df7d8f-cert\") pod \"ingress-canary-mfdxs\" (UID: \"4c93581a-4009-4914-b149-b586d7df7d8f\") " pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.066142 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.066042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb62ad6a-225c-4bb0-8421-f79008e5b666-tmp-dir\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.066142 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.066111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdlf\" (UniqueName: \"kubernetes.io/projected/4c93581a-4009-4914-b149-b586d7df7d8f-kube-api-access-wbdlf\") pod \"ingress-canary-mfdxs\" (UID: \"4c93581a-4009-4914-b149-b586d7df7d8f\") " pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.066242 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.066141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.066242 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.066160    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpht\" (UniqueName: \"kubernetes.io/projected/bb62ad6a-225c-4bb0-8421-f79008e5b666-kube-api-access-wvpht\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.066242 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.066187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb62ad6a-225c-4bb0-8421-f79008e5b666-config-volume\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.123217 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.123008    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:21.166865 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.166840    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c93581a-4009-4914-b149-b586d7df7d8f-cert\") pod \"ingress-canary-mfdxs\" (UID: \"4c93581a-4009-4914-b149-b586d7df7d8f\") " pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.166933 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.166881    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb62ad6a-225c-4bb0-8421-f79008e5b666-tmp-dir\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.166933 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.166911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdlf\" (UniqueName: \"kubernetes.io/projected/4c93581a-4009-4914-b149-b586d7df7d8f-kube-api-access-wbdlf\") pod \"ingress-canary-mfdxs\" (UID: \"4c93581a-4009-4914-b149-b586d7df7d8f\") " pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.166933 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.166930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.167062 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.166944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpht\" (UniqueName: \"kubernetes.io/projected/bb62ad6a-225c-4bb0-8421-f79008e5b666-kube-api-access-wvpht\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.167062 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.166960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb62ad6a-225c-4bb0-8421-f79008e5b666-config-volume\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.167158 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:21.167139    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:21.167241 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:21.167228    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:21.667211297 +0000 UTC m=+35.630826250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:21.167329 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.167306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb62ad6a-225c-4bb0-8421-f79008e5b666-tmp-dir\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.167506 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.167489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb62ad6a-225c-4bb0-8421-f79008e5b666-config-volume\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.170787 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.170768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c93581a-4009-4914-b149-b586d7df7d8f-cert\") pod \"ingress-canary-mfdxs\" (UID: \"4c93581a-4009-4914-b149-b586d7df7d8f\") " pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.175719 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.175697    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdlf\" (UniqueName: \"kubernetes.io/projected/4c93581a-4009-4914-b149-b586d7df7d8f-kube-api-access-wbdlf\") pod \"ingress-canary-mfdxs\" (UID: \"4c93581a-4009-4914-b149-b586d7df7d8f\") " pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.176003 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.175984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpht\" (UniqueName: \"kubernetes.io/projected/bb62ad6a-225c-4bb0-8421-f79008e5b666-kube-api-access-wvpht\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.253189 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.253158    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfdxs"
Sep 11 13:58:21.497694 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.497530    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:21.497838 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.497586    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:21.497914 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.497586    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:21.500422 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.500400    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:58:21.500654 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.500628    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgrdt\""
Sep 11 13:58:21.500760 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.500662    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:58:21.500760 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.500628    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:58:21.500760 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.500683    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:58:21.501379 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.501354    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh858\""
Sep 11 13:58:21.566092 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.566061    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfdxs"]
Sep 11 13:58:21.569179 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:58:21.569155    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c93581a_4009_4914_b149_b586d7df7d8f.slice/crio-46c979c762e152374e03a9aabd457a20770f5288645238c6e2b0e61857289bdd WatchSource:0}: Error finding container 46c979c762e152374e03a9aabd457a20770f5288645238c6e2b0e61857289bdd: Status 404 returned error can't find the container with id 46c979c762e152374e03a9aabd457a20770f5288645238c6e2b0e61857289bdd
Sep 11 13:58:21.671288 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.671259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:21.671412 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:21.671363    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:21.671453 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:21.671421    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:22.671407369 +0000 UTC m=+36.635022321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:21.746624 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.746598    2558 generic.go:356] "Generic (PLEG): container finished" podID="b798c22e-4497-47e4-826a-a54445aef97f" containerID="0902efa48060bf72c2d5930f5e1707a1048bc542c43cf4f73e76fb32da77fcc3" exitCode=0
Sep 11 13:58:21.746749 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.746674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerDied","Data":"0902efa48060bf72c2d5930f5e1707a1048bc542c43cf4f73e76fb32da77fcc3"}
Sep 11 13:58:21.748007 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:21.747947    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfdxs" event={"ID":"4c93581a-4009-4914-b149-b586d7df7d8f","Type":"ContainerStarted","Data":"46c979c762e152374e03a9aabd457a20770f5288645238c6e2b0e61857289bdd"}
Sep 11 13:58:22.679294 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:22.679256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:22.680003 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:22.679435    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:22.680003 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:22.679511    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:24.679490074 +0000 UTC m=+38.643105028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:22.754347 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:22.754309    2558 generic.go:356] "Generic (PLEG): container finished" podID="b798c22e-4497-47e4-826a-a54445aef97f" containerID="41f8b528c10c9839f52e7e762f3cc5116f616be22523c4e97d8392d58ebba753" exitCode=0
Sep 11 13:58:22.754487 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:22.754360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerDied","Data":"41f8b528c10c9839f52e7e762f3cc5116f616be22523c4e97d8392d58ebba753"}
Sep 11 13:58:23.764179 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:23.764134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" event={"ID":"b798c22e-4497-47e4-826a-a54445aef97f","Type":"ContainerStarted","Data":"2d8675a395ce30b39f3446d56da73b522247cb30429e715acae0a87d22fcd247"}
Sep 11 13:58:23.813719 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:23.813662    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sjdj6" podStartSLOduration=7.642360033 podStartE2EDuration="36.813643682s" totalImagesPullingTime="29.171283649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:23.812469652 +0000 UTC m=+37.776084626" watchObservedRunningTime="2026-09-11 13:58:23.813643682 +0000 UTC m=+37.777258657"
Sep 11 13:58:23.950662 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:23.950633    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:24.053639 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:24.053517    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:24.696152 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:24.696120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:24.696296 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:24.696237    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:24.696296 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:24.696291    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:28.696276665 +0000 UTC m=+42.659891616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:24.768088 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:24.768059    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfdxs" event={"ID":"4c93581a-4009-4914-b149-b586d7df7d8f","Type":"ContainerStarted","Data":"91e77899ca00d1ddd475b670899b1ad523483b4aaff89dd6ede5a76bc60a58b3"}
Sep 11 13:58:24.787199 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:24.787164    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mfdxs" podStartSLOduration=2.380605327 podStartE2EDuration="4.787151199s" totalImagesPullingTime="2.406545872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:24.786051506 +0000 UTC m=+38.749666473" watchObservedRunningTime="2026-09-11 13:58:24.787151199 +0000 UTC m=+38.750766192"
Sep 11 13:58:25.238298 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:25.238269    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:25.300917 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:25.300875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:28.720805 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:28.720771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:28.721230 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:28.720923    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:28.721230 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:28.721001    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:36.720983104 +0000 UTC m=+50.684598080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:30.663160 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:30.663128    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:30.730691 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:30.730659    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:36.771018 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:36.770979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:36.771558 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:36.771123    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:36.771558 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:36.771187    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:52.771170904 +0000 UTC m=+66.734785856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:38.574632 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:38.574583    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:38.679890 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:38.679860    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:45.737678 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:45.737650    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsrq8"
Sep 11 13:58:45.901813 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:45.901776    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:45.975471 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:45.975441    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:49.319776 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.319739    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7956fdcb7d-pftq7"]
Sep 11 13:58:49.322568 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.322554    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.326149 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.326115    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 13:58:49.326283 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.326128    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 13:58:49.326283 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.326194    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 13:58:49.330323 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.330305    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mdw98\""
Sep 11 13:58:49.330678 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.330663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 13:58:49.330861 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.330847    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 13:58:49.332329 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.331654    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 13:58:49.337738 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.337708    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7956fdcb7d-pftq7"]
Sep 11 13:58:49.420373 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.420330    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"]
Sep 11 13:58:49.422270 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.422252    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:49.425263 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.425245    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jvfsg\""
Sep 11 13:58:49.425357 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.425344    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:58:49.425419 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.425345    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 13:58:49.425419 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.425407    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 13:58:49.435134 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.435112    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"]
Sep 11 13:58:49.448452 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.448434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-default-certificate\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.448584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.448459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.448584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.448476    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.448584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.448493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-stats-auth\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.448584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.448575    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshx4\" (UniqueName: \"kubernetes.io/projected/d14d4b46-4404-43d0-80b2-1661626901cb-kube-api-access-bshx4\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.497402 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.497377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:49.549200 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-default-certificate\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.549269 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.549269 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549219    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.549269 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-stats-auth\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.549408 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:49.549320    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:58:49.549459 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549410    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshx4\" (UniqueName: \"kubernetes.io/projected/d14d4b46-4404-43d0-80b2-1661626901cb-kube-api-access-bshx4\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.549459 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:49.549452    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:50.049431507 +0000 UTC m=+64.013046464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : secret "router-metrics-certs-default" not found
Sep 11 13:58:49.549561 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2sm\" (UniqueName: \"kubernetes.io/projected/521e8c3d-2e52-4224-9845-ceb8745b80b8-kube-api-access-rr2sm\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:49.549561 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:49.549515    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:50.049497724 +0000 UTC m=+64.013112689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : configmap references non-existent config key: service-ca.crt
Sep 11 13:58:49.549561 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.549555    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:49.552441 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.552417    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-stats-auth\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.552619 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.552599    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-default-certificate\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.557603 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.557585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshx4\" (UniqueName: \"kubernetes.io/projected/d14d4b46-4404-43d0-80b2-1661626901cb-kube-api-access-bshx4\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:49.562821 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.562790    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:49.649881 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.649820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2sm\" (UniqueName: \"kubernetes.io/projected/521e8c3d-2e52-4224-9845-ceb8745b80b8-kube-api-access-rr2sm\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:49.649881 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.649858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:49.650023 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:49.650005    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 13:58:49.650090 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:49.650069    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls podName:521e8c3d-2e52-4224-9845-ceb8745b80b8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:50.150055559 +0000 UTC m=+64.113670516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8kg55" (UID: "521e8c3d-2e52-4224-9845-ceb8745b80b8") : secret "samples-operator-tls" not found
Sep 11 13:58:49.660055 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:49.660020    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2sm\" (UniqueName: \"kubernetes.io/projected/521e8c3d-2e52-4224-9845-ceb8745b80b8-kube-api-access-rr2sm\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:50.052225 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:50.052201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:50.052346 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:50.052231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:50.052391 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:50.052346    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:58:50.052438 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:50.052425    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.052402698 +0000 UTC m=+65.016017664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : configmap references non-existent config key: service-ca.crt
Sep 11 13:58:50.052488 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:50.052446    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.052438132 +0000 UTC m=+65.016053083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : secret "router-metrics-certs-default" not found
Sep 11 13:58:50.152960 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:50.152936    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:50.153095 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:50.153077    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 13:58:50.153144 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:50.153135    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls podName:521e8c3d-2e52-4224-9845-ceb8745b80b8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.153121136 +0000 UTC m=+65.116736093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8kg55" (UID: "521e8c3d-2e52-4224-9845-ceb8745b80b8") : secret "samples-operator-tls" not found
Sep 11 13:58:51.059425 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.059380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:51.059425 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.059425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:51.059898 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.059521    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:58:51.059898 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.059551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:53.059535976 +0000 UTC m=+67.023150928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : configmap references non-existent config key: service-ca.crt
Sep 11 13:58:51.059898 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.059570    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:53.059558841 +0000 UTC m=+67.023173798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : secret "router-metrics-certs-default" not found
Sep 11 13:58:51.160531 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.160497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:51.160702 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.160645    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 13:58:51.160756 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.160707    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls podName:521e8c3d-2e52-4224-9845-ceb8745b80b8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:53.160693529 +0000 UTC m=+67.124308482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8kg55" (UID: "521e8c3d-2e52-4224-9845-ceb8745b80b8") : secret "samples-operator-tls" not found
Sep 11 13:58:51.261196 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.261170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:51.261314 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.261215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:58:51.263947 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.263926    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:58:51.264041 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.264008    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:58:51.271701 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.271686    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 13:58:51.271754 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:51.271735    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs podName:dd187b1d-bbab-45a2-88a3-1d32dec00e36 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:55.271721507 +0000 UTC m=+129.235336464 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs") pod "network-metrics-daemon-th28r" (UID: "dd187b1d-bbab-45a2-88a3-1d32dec00e36") : secret "metrics-daemon-secret" not found
Sep 11 13:58:51.273825 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.273809    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04cb311f-1d53-4152-9f5d-66fb2e50ca01-original-pull-secret\") pod \"global-pull-secret-syncer-qtbqd\" (UID: \"04cb311f-1d53-4152-9f5d-66fb2e50ca01\") " pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:51.362273 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.362223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:51.364985 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.364968    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:58:51.375177 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.375158    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:58:51.385491 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.385472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7s5z\" (UniqueName: \"kubernetes.io/projected/f87e021d-43bf-40e2-b0bd-d251c9c19079-kube-api-access-x7s5z\") pod \"network-check-target-bqdgz\" (UID: \"f87e021d-43bf-40e2-b0bd-d251c9c19079\") " pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:51.507863 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.507841    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qtbqd"
Sep 11 13:58:51.514899 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.514878    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgrdt\""
Sep 11 13:58:51.523853 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.523836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:51.585142 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.585114    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:51.686556 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.686492    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:51.773280 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.773253    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qtbqd"]
Sep 11 13:58:51.775855 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:58:51.775834    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cb311f_1d53_4152_9f5d_66fb2e50ca01.slice/crio-0ff1e3d0a7217b2a1d93b9fd487c9eda862e1edee824bd55243ef04a57a2e7d7 WatchSource:0}: Error finding container 0ff1e3d0a7217b2a1d93b9fd487c9eda862e1edee824bd55243ef04a57a2e7d7: Status 404 returned error can't find the container with id 0ff1e3d0a7217b2a1d93b9fd487c9eda862e1edee824bd55243ef04a57a2e7d7
Sep 11 13:58:51.796835 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.796812    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bqdgz"]
Sep 11 13:58:51.799563 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:58:51.799538    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87e021d_43bf_40e2_b0bd_d251c9c19079.slice/crio-152a2fbbb610abeb3dc1fd3e68885dc792913ec8208e8a9bb27f079cca9952a3 WatchSource:0}: Error finding container 152a2fbbb610abeb3dc1fd3e68885dc792913ec8208e8a9bb27f079cca9952a3: Status 404 returned error can't find the container with id 152a2fbbb610abeb3dc1fd3e68885dc792913ec8208e8a9bb27f079cca9952a3
Sep 11 13:58:51.830283 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.830255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bqdgz" event={"ID":"f87e021d-43bf-40e2-b0bd-d251c9c19079","Type":"ContainerStarted","Data":"152a2fbbb610abeb3dc1fd3e68885dc792913ec8208e8a9bb27f079cca9952a3"}
Sep 11 13:58:51.831286 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:51.831265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qtbqd" event={"ID":"04cb311f-1d53-4152-9f5d-66fb2e50ca01","Type":"ContainerStarted","Data":"0ff1e3d0a7217b2a1d93b9fd487c9eda862e1edee824bd55243ef04a57a2e7d7"}
Sep 11 13:58:52.772517 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:52.772479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:58:52.772963 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:52.772643    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:58:52.772963 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:52.772707    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls podName:bb62ad6a-225c-4bb0-8421-f79008e5b666 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:24.772688494 +0000 UTC m=+98.736303448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls") pod "dns-default-94z9z" (UID: "bb62ad6a-225c-4bb0-8421-f79008e5b666") : secret "dns-default-metrics-tls" not found
Sep 11 13:58:53.074741 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:53.074517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:53.074741 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:53.074565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:53.074741 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:53.074676    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:58:53.074741 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:53.074723    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:57.074703687 +0000 UTC m=+71.038318643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : configmap references non-existent config key: service-ca.crt
Sep 11 13:58:53.075069 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:53.074748    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:58:57.074734612 +0000 UTC m=+71.038349565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : secret "router-metrics-certs-default" not found
Sep 11 13:58:53.175615 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:53.175576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:53.175781 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:53.175762    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 13:58:53.175835 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:53.175826    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls podName:521e8c3d-2e52-4224-9845-ceb8745b80b8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:57.175809008 +0000 UTC m=+71.139423962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8kg55" (UID: "521e8c3d-2e52-4224-9845-ceb8745b80b8") : secret "samples-operator-tls" not found
Sep 11 13:58:53.835801 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:53.835704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:54.319978 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.319941    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c8rm7"]
Sep 11 13:58:54.332818 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.332783    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:54.335668 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.335647    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.339790 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.339767    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 13:58:54.340284 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.340262    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 13:58:54.341162 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.340784    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 13:58:54.341162 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.340911    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c8rm7"]
Sep 11 13:58:54.341162 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.340987    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 13:58:54.342741 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.342722    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-85l57\""
Sep 11 13:58:54.387043 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.386990    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgc5\" (UniqueName: \"kubernetes.io/projected/fdf104d2-6dd5-4717-ba07-687b56918990-kube-api-access-ldgc5\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.387232 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.387080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.387232 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.387135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdf104d2-6dd5-4717-ba07-687b56918990-crio-socket\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.387232 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.387176    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdf104d2-6dd5-4717-ba07-687b56918990-data-volume\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.387384 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.387234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.426340 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.426303    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:54.488561 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.488522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdf104d2-6dd5-4717-ba07-687b56918990-data-volume\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.488561 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.488563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.488795 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.488624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgc5\" (UniqueName: \"kubernetes.io/projected/fdf104d2-6dd5-4717-ba07-687b56918990-kube-api-access-ldgc5\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.488795 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.488663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.488795 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.488776    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:54.488943 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.488826    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:58:54.488943 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.488884    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdf104d2-6dd5-4717-ba07-687b56918990-data-volume\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.488943 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.488839    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:54.988825421 +0000 UTC m=+68.952440373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:54.489121 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.488960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdf104d2-6dd5-4717-ba07-687b56918990-crio-socket\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.489121 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.489015    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdf104d2-6dd5-4717-ba07-687b56918990-crio-socket\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.489121 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.489048    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:54.988995455 +0000 UTC m=+68.952610415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : configmap "kube-rbac-proxy" not found
Sep 11 13:58:54.498769 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.498744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgc5\" (UniqueName: \"kubernetes.io/projected/fdf104d2-6dd5-4717-ba07-687b56918990-kube-api-access-ldgc5\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.993438 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.993404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.993792 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:54.993465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:54.993792 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.993528    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:58:54.993792 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.993550    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:54.993792 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.993591    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:55.993577687 +0000 UTC m=+69.957192638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : configmap "kube-rbac-proxy" not found
Sep 11 13:58:54.993792 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:54.993603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:55.993597826 +0000 UTC m=+69.957212778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:56.000559 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.000522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:56.001060 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.000632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:56.001060 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:56.000666    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:58:56.001060 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:56.000737    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:58.00072124 +0000 UTC m=+71.964336224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : configmap "kube-rbac-proxy" not found
Sep 11 13:58:56.001060 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:56.000785    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:56.001060 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:56.000842    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:58.000826125 +0000 UTC m=+71.964441112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:56.225134 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.225104    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mxh4_c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2/dns-node-resolver/0.log"
Sep 11 13:58:56.374929 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.374861    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"]
Sep 11 13:58:56.396770 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.396746    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"]
Sep 11 13:58:56.396889 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.396836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"
Sep 11 13:58:56.401778 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.401757    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p46m8\""
Sep 11 13:58:56.505079 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.505056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqtx\" (UniqueName: \"kubernetes.io/projected/1afbe8df-57b5-4e6d-be27-33e475162b90-kube-api-access-srqtx\") pod \"network-check-source-6f8b768455-mvpvl\" (UID: \"1afbe8df-57b5-4e6d-be27-33e475162b90\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"
Sep 11 13:58:56.605387 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.605360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqtx\" (UniqueName: \"kubernetes.io/projected/1afbe8df-57b5-4e6d-be27-33e475162b90-kube-api-access-srqtx\") pod \"network-check-source-6f8b768455-mvpvl\" (UID: \"1afbe8df-57b5-4e6d-be27-33e475162b90\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"
Sep 11 13:58:56.619211 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.619185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqtx\" (UniqueName: \"kubernetes.io/projected/1afbe8df-57b5-4e6d-be27-33e475162b90-kube-api-access-srqtx\") pod \"network-check-source-6f8b768455-mvpvl\" (UID: \"1afbe8df-57b5-4e6d-be27-33e475162b90\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"
Sep 11 13:58:56.704806 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.704780    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"
Sep 11 13:58:56.778353 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.778327    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:56.851333 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:56.851137    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:57.025402 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.025336    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wjfc_d8eb0a86-1eea-4d55-87a7-1bca0e1ed936/node-ca/0.log"
Sep 11 13:58:57.071130 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.071100    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl"]
Sep 11 13:58:57.074249 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:58:57.074224    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afbe8df_57b5_4e6d_be27_33e475162b90.slice/crio-3ec910994fc7e9754aabffa3112b39a19b9486a5c661c4dbc8327385fcf1a02e WatchSource:0}: Error finding container 3ec910994fc7e9754aabffa3112b39a19b9486a5c661c4dbc8327385fcf1a02e: Status 404 returned error can't find the container with id 3ec910994fc7e9754aabffa3112b39a19b9486a5c661c4dbc8327385fcf1a02e
Sep 11 13:58:57.109849 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.109823    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:57.109949 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.109862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:58:57.110007 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:57.109963    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:58:57.110082 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:57.110022    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:59:05.110003494 +0000 UTC m=+79.073618445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : secret "router-metrics-certs-default" not found
Sep 11 13:58:57.110082 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:57.110060    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:59:05.110051062 +0000 UTC m=+79.073666014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : configmap references non-existent config key: service-ca.crt
Sep 11 13:58:57.210516 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.210487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:58:57.210647 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:57.210612    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 13:58:57.210687 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:57.210659    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls podName:521e8c3d-2e52-4224-9845-ceb8745b80b8 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:05.210647274 +0000 UTC m=+79.174262226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8kg55" (UID: "521e8c3d-2e52-4224-9845-ceb8745b80b8") : secret "samples-operator-tls" not found
Sep 11 13:58:57.772736 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.772707    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"]
Sep 11 13:58:57.774760 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.774742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"
Sep 11 13:58:57.777930 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.777911    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4mpcv\""
Sep 11 13:58:57.778476 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.778460    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 13:58:57.778566 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.778494    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 13:58:57.784911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.784891    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"]
Sep 11 13:58:57.814367 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.814343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srrt\" (UniqueName: \"kubernetes.io/projected/4d2dc9d3-1a28-4e10-a355-8a22134d9cc9-kube-api-access-6srrt\") pod \"migrator-8b4d697c5-7fdsd\" (UID: \"4d2dc9d3-1a28-4e10-a355-8a22134d9cc9\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"
Sep 11 13:58:57.819188 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.819162    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:57.849913 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.849889    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qtbqd" event={"ID":"04cb311f-1d53-4152-9f5d-66fb2e50ca01","Type":"ContainerStarted","Data":"f8d62bbafcbb930ad220ad2a0a9005a26d8ecd2fcc2b5a41562705f94ef67b34"}
Sep 11 13:58:57.851429 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.851410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bqdgz" event={"ID":"f87e021d-43bf-40e2-b0bd-d251c9c19079","Type":"ContainerStarted","Data":"1aaece424fc24e66d627c1a97a2072f1b4eb59db786955e906d113562a99e7c9"}
Sep 11 13:58:57.851528 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.851462    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:58:57.852944 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.852925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl" event={"ID":"1afbe8df-57b5-4e6d-be27-33e475162b90","Type":"ContainerStarted","Data":"8ea75a752eeb2554fdbb2b50ca5be9b4d9758fe0c3fc1b8143f5fbb381d35c39"}
Sep 11 13:58:57.853101 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.852954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl" event={"ID":"1afbe8df-57b5-4e6d-be27-33e475162b90","Type":"ContainerStarted","Data":"3ec910994fc7e9754aabffa3112b39a19b9486a5c661c4dbc8327385fcf1a02e"}
Sep 11 13:58:57.867471 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.867428    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qtbqd" podStartSLOduration=65.828677147 podStartE2EDuration="1m10.867416268s" totalImagesPullingTime="5.038739121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:57.86664363 +0000 UTC m=+71.830258603" watchObservedRunningTime="2026-09-11 13:58:57.867416268 +0000 UTC m=+71.831031233"
Sep 11 13:58:57.884463 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.884431    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mvpvl" podStartSLOduration=1.884420601 podStartE2EDuration="1.884420601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:57.883573544 +0000 UTC m=+71.847188528" watchObservedRunningTime="2026-09-11 13:58:57.884420601 +0000 UTC m=+71.848035575"
Sep 11 13:58:57.902635 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.902599    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bqdgz" podStartSLOduration=65.899990337 podStartE2EDuration="1m10.902588162s" totalImagesPullingTime="5.002597825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:57.901870999 +0000 UTC m=+71.865485974" watchObservedRunningTime="2026-09-11 13:58:57.902588162 +0000 UTC m=+71.866203136"
Sep 11 13:58:57.908266 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.908232    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:58:57.915255 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.915232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srrt\" (UniqueName: \"kubernetes.io/projected/4d2dc9d3-1a28-4e10-a355-8a22134d9cc9-kube-api-access-6srrt\") pod \"migrator-8b4d697c5-7fdsd\" (UID: \"4d2dc9d3-1a28-4e10-a355-8a22134d9cc9\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"
Sep 11 13:58:57.930504 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:57.930480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srrt\" (UniqueName: \"kubernetes.io/projected/4d2dc9d3-1a28-4e10-a355-8a22134d9cc9-kube-api-access-6srrt\") pod \"migrator-8b4d697c5-7fdsd\" (UID: \"4d2dc9d3-1a28-4e10-a355-8a22134d9cc9\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"
Sep 11 13:58:58.015782 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:58.015756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:58.015909 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:58.015866    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:58:58.015909 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:58.015898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:58:58.016049 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:58.015917    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:02.015904889 +0000 UTC m=+75.979519846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : configmap "kube-rbac-proxy" not found
Sep 11 13:58:58.016049 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:58.015962    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:58.016049 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:58:58.016017    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:02.016000224 +0000 UTC m=+75.979615176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:58:58.083527 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:58.083233    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"
Sep 11 13:58:58.225746 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:58.225716    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfdxs_4c93581a-4009-4914-b149-b586d7df7d8f/serve-healthcheck-canary/0.log"
Sep 11 13:58:58.355158 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:58.355108    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd"]
Sep 11 13:58:58.357946 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:58:58.357924    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2dc9d3_1a28_4e10_a355_8a22134d9cc9.slice/crio-c1f26517d0c02a8fbacf947298c49744bee4dd2efa26dac76c2cb6c2241b629f WatchSource:0}: Error finding container c1f26517d0c02a8fbacf947298c49744bee4dd2efa26dac76c2cb6c2241b629f: Status 404 returned error can't find the container with id c1f26517d0c02a8fbacf947298c49744bee4dd2efa26dac76c2cb6c2241b629f
Sep 11 13:58:58.860544 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:58.860423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd" event={"ID":"4d2dc9d3-1a28-4e10-a355-8a22134d9cc9","Type":"ContainerStarted","Data":"c1f26517d0c02a8fbacf947298c49744bee4dd2efa26dac76c2cb6c2241b629f"}
Sep 11 13:58:59.864930 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:59.864895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd" event={"ID":"4d2dc9d3-1a28-4e10-a355-8a22134d9cc9","Type":"ContainerStarted","Data":"d8dbd2483091abe45f0d72f59b8fab28afa92b7ecbde62234b196a3e45eebe1f"}
Sep 11 13:58:59.865374 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:59.864992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd" event={"ID":"4d2dc9d3-1a28-4e10-a355-8a22134d9cc9","Type":"ContainerStarted","Data":"78952d070650f7b186c48a0ae72b57d8adb28805da61a127ee292654869f5bbf"}
Sep 11 13:58:59.887160 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:58:59.887118    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7fdsd" podStartSLOduration=1.865054481 podStartE2EDuration="2.88710365s" totalImagesPullingTime="1.022049169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:59.886382456 +0000 UTC m=+73.849997430" watchObservedRunningTime="2026-09-11 13:58:59.88710365 +0000 UTC m=+73.850718670"
Sep 11 13:59:00.039523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.039492    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:00.136839 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.136769    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:00.296054 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.296000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:00.402077 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.402008    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:00.775460 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.775426    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:00.877212 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.877179    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:00.994106 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:00.994078    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:01.090005 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:01.089931    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:02.044883 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:02.044838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:02.045356 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:02.044912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:02.045356 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:02.044985    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:59:02.045356 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:02.045060    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:10.045045962 +0000 UTC m=+84.008660914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:59:02.045356 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:02.044989    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:59:02.045356 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:02.045148    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:10.045126586 +0000 UTC m=+84.008741543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : configmap "kube-rbac-proxy" not found
Sep 11 13:59:02.662757 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:02.662720    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:02.784114 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:02.784078    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:04.657439 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:04.657401    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:04.859265 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:04.859232    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:04.984997 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:04.984973    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:05.167862 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:05.167829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:05.167862 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:05.167863    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:05.168066 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:05.167974    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:59:05.168066 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:05.168001    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:59:21.167984297 +0000 UTC m=+95.131599249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : configmap references non-existent config key: service-ca.crt
Sep 11 13:59:05.168066 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:05.168051    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs podName:d14d4b46-4404-43d0-80b2-1661626901cb nodeName:}" failed. No retries permitted until 2026-09-11 13:59:21.168020144 +0000 UTC m=+95.131635100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs") pod "router-default-7956fdcb7d-pftq7" (UID: "d14d4b46-4404-43d0-80b2-1661626901cb") : secret "router-metrics-certs-default" not found
Sep 11 13:59:05.173912 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:05.173888    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:05.269001 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:05.268932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:59:05.269126 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:05.269088    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 13:59:05.269168 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:05.269157    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls podName:521e8c3d-2e52-4224-9845-ceb8745b80b8 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:21.269140636 +0000 UTC m=+95.232755587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8kg55" (UID: "521e8c3d-2e52-4224-9845-ceb8745b80b8") : secret "samples-operator-tls" not found
Sep 11 13:59:06.166540 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:06.166505    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:06.577486 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:06.577441    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:06.927193 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:06.927099    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:07.049063 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:07.049017    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:07.214539 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:07.214509    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:07.536145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:07.536053    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:07.922439 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:07.922341    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:08.192280 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:08.192245    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:08.339316 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:08.339279    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:08.453650 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:08.453579    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:08.782103 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:08.781994    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:08.973021 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:08.972918    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:09.084396 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:09.084335    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:09.183825 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:09.183802    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:10.103171 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.103144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:10.103537 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.103207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:10.103537 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:10.103314    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:59:10.103537 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:10.103396    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls podName:fdf104d2-6dd5-4717-ba07-687b56918990 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:26.103377304 +0000 UTC m=+100.066992256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c8rm7" (UID: "fdf104d2-6dd5-4717-ba07-687b56918990") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:59:10.103805 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.103781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdf104d2-6dd5-4717-ba07-687b56918990-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:10.382987 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.382900    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:10.531746 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.531711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:10.681125 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.681044    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:10.860042 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:10.860000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:11.040900 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:11.040870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:11.164827 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:11.164800    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:11.759748 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:11.759716    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:11.862238 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:11.862201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:12.360928 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:12.360900    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:12.556525 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:12.556493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:12.788607 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:12.788575    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:13.073877 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:13.073775    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:13.185121 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:13.185093    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:13.970998 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:13.970954    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:14.354128 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:14.354045    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:14.549601 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:14.549575    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:14.947358 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:14.947305    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:15.142574 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:15.142539    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:16.704553 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:16.704514    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:21.179903 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.179862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:21.179903 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.179904    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:21.180490 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.180470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d14d4b46-4404-43d0-80b2-1661626901cb-service-ca-bundle\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:21.182215 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.182195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d4b46-4404-43d0-80b2-1661626901cb-metrics-certs\") pod \"router-default-7956fdcb7d-pftq7\" (UID: \"d14d4b46-4404-43d0-80b2-1661626901cb\") " pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:21.281095 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.281069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:59:21.283307 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.283289    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/521e8c3d-2e52-4224-9845-ceb8745b80b8-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8kg55\" (UID: \"521e8c3d-2e52-4224-9845-ceb8745b80b8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:59:21.431230 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.431174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:21.530774 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.530746    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"
Sep 11 13:59:21.710088 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.710055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7956fdcb7d-pftq7"]
Sep 11 13:59:21.712620 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:21.712590    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14d4b46_4404_43d0_80b2_1661626901cb.slice/crio-3d379ace5ecba70a41ac57df6fa8f0bd072ac34cbfdc6b43634fcbc3ff42aaa4 WatchSource:0}: Error finding container 3d379ace5ecba70a41ac57df6fa8f0bd072ac34cbfdc6b43634fcbc3ff42aaa4: Status 404 returned error can't find the container with id 3d379ace5ecba70a41ac57df6fa8f0bd072ac34cbfdc6b43634fcbc3ff42aaa4
Sep 11 13:59:21.799148 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.799120    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55"]
Sep 11 13:59:21.933153 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.933123    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55" event={"ID":"521e8c3d-2e52-4224-9845-ceb8745b80b8","Type":"ContainerStarted","Data":"b5c01122ba7d1fecb9f5165b756423ed0fa958cf58b29b09727e0e74c5724f71"}
Sep 11 13:59:21.934769 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.934742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7956fdcb7d-pftq7" event={"ID":"d14d4b46-4404-43d0-80b2-1661626901cb","Type":"ContainerStarted","Data":"91a24c49831874d33000d2049b4bd5f36dd019fa886cffc7d454bf1ff47310ac"}
Sep 11 13:59:21.934864 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.934781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7956fdcb7d-pftq7" event={"ID":"d14d4b46-4404-43d0-80b2-1661626901cb","Type":"ContainerStarted","Data":"3d379ace5ecba70a41ac57df6fa8f0bd072ac34cbfdc6b43634fcbc3ff42aaa4"}
Sep 11 13:59:21.955553 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:21.955515    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7956fdcb7d-pftq7" podStartSLOduration=32.955502216 podStartE2EDuration="32.955502216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:21.954647275 +0000 UTC m=+95.918262249" watchObservedRunningTime="2026-09-11 13:59:21.955502216 +0000 UTC m=+95.919117189"
Sep 11 13:59:22.431753 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:22.431724    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:22.434133 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:22.434112    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:22.938709 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:22.938680    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:22.939990 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:22.939967    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7956fdcb7d-pftq7"
Sep 11 13:59:23.303817 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.303780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:23.382697 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.382658    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:23.471193 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.471164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:23.820524 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.820453    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:23.943604 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.943561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55" event={"ID":"521e8c3d-2e52-4224-9845-ceb8745b80b8","Type":"ContainerStarted","Data":"07b84e4c644681b8cbf6d16000953a026439580d6d80a6c1d13dbfc98309303c"}
Sep 11 13:59:23.943727 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.943615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55" event={"ID":"521e8c3d-2e52-4224-9845-ceb8745b80b8","Type":"ContainerStarted","Data":"d8f552900419bf9adcc55b6dc361a735ad29caeeacad018e6c97946912f5edfc"}
Sep 11 13:59:23.973662 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:23.973609    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8kg55" podStartSLOduration=33.284559925 podStartE2EDuration="34.973592846s" totalImagesPullingTime="1.689032921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:23.973568213 +0000 UTC m=+97.937183187" watchObservedRunningTime="2026-09-11 13:59:23.973592846 +0000 UTC m=+97.937207820"
Sep 11 13:59:24.813472 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:24.813433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:59:24.815903 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:24.815867    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb62ad6a-225c-4bb0-8421-f79008e5b666-metrics-tls\") pod \"dns-default-94z9z\" (UID: \"bb62ad6a-225c-4bb0-8421-f79008e5b666\") " pod="openshift-dns/dns-default-94z9z"
Sep 11 13:59:24.828235 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:24.828213    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tbm52\""
Sep 11 13:59:24.836330 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:24.836315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94z9z"
Sep 11 13:59:25.107011 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:25.106951    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94z9z"]
Sep 11 13:59:25.111160 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:25.111140    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb62ad6a_225c_4bb0_8421_f79008e5b666.slice/crio-a55e4d2d4ee934c4b194eaf965be3427f56718aa4853868121bebec0587bc67d WatchSource:0}: Error finding container a55e4d2d4ee934c4b194eaf965be3427f56718aa4853868121bebec0587bc67d: Status 404 returned error can't find the container with id a55e4d2d4ee934c4b194eaf965be3427f56718aa4853868121bebec0587bc67d
Sep 11 13:59:25.950900 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:25.950864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94z9z" event={"ID":"bb62ad6a-225c-4bb0-8421-f79008e5b666","Type":"ContainerStarted","Data":"a55e4d2d4ee934c4b194eaf965be3427f56718aa4853868121bebec0587bc67d"}
Sep 11 13:59:26.122189 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.122158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:26.124357 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.124335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdf104d2-6dd5-4717-ba07-687b56918990-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c8rm7\" (UID: \"fdf104d2-6dd5-4717-ba07-687b56918990\") " pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:26.147248 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.147225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c8rm7"
Sep 11 13:59:26.431559 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.431525    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c8rm7"]
Sep 11 13:59:26.717176 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.717150    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:26.723080 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:26.723016    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf104d2_6dd5_4717_ba07_687b56918990.slice/crio-8c0b6a54449fc4f6974b29d0680c1fdda1cfed0f418c312396ed293ac37a40b0 WatchSource:0}: Error finding container 8c0b6a54449fc4f6974b29d0680c1fdda1cfed0f418c312396ed293ac37a40b0: Status 404 returned error can't find the container with id 8c0b6a54449fc4f6974b29d0680c1fdda1cfed0f418c312396ed293ac37a40b0
Sep 11 13:59:26.818478 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.818442    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:26.955724 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.955696    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94z9z" event={"ID":"bb62ad6a-225c-4bb0-8421-f79008e5b666","Type":"ContainerStarted","Data":"6e9bdfc0b76256ef6e70f6d0ace379d3f435abd2e9ff84635e16014c3b23382a"}
Sep 11 13:59:26.956098 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.955747    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94z9z" event={"ID":"bb62ad6a-225c-4bb0-8421-f79008e5b666","Type":"ContainerStarted","Data":"51478fb56a4cbda0ca33dfbf35e1110ef97cdea46fcc25ca6132d5fcdbfc2f46"}
Sep 11 13:59:26.956098 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.955798    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94z9z"
Sep 11 13:59:26.957715 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.957689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c8rm7" event={"ID":"fdf104d2-6dd5-4717-ba07-687b56918990","Type":"ContainerStarted","Data":"924fe40e0aaba9696fc534ace75b729179e188015863de67159dcc4306376f6c"}
Sep 11 13:59:26.957799 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.957737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c8rm7" event={"ID":"fdf104d2-6dd5-4717-ba07-687b56918990","Type":"ContainerStarted","Data":"8c0b6a54449fc4f6974b29d0680c1fdda1cfed0f418c312396ed293ac37a40b0"}
Sep 11 13:59:26.985407 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:26.985341    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-94z9z" podStartSLOduration=65.323686866 podStartE2EDuration="1m6.985328246s" totalImagesPullingTime="1.66164138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:26.984750546 +0000 UTC m=+100.948365522" watchObservedRunningTime="2026-09-11 13:59:26.985328246 +0000 UTC m=+100.948943219"
Sep 11 13:59:27.962435 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:27.962396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c8rm7" event={"ID":"fdf104d2-6dd5-4717-ba07-687b56918990","Type":"ContainerStarted","Data":"a502f697789f1c21af137ec9f11face344dcb2bff9df88df9d6846a47cef9147"}
Sep 11 13:59:28.079543 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.079505    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7686b6bc78-kcjfj"]
Sep 11 13:59:28.082683 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.082651    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.086217 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.086195    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kzr4h\""
Sep 11 13:59:28.087136 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.087114    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 13:59:28.087136 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.087127    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 13:59:28.087305 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.087165    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 13:59:28.098209 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.098189    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 13:59:28.107829 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.107776    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7686b6bc78-kcjfj"]
Sep 11 13:59:28.137764 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-installation-pull-secrets\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.137868 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-registry-tls\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.137868 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-ca-trust-extracted\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.137868 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-image-registry-private-configuration\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.138023 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137904    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hs7d\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-kube-api-access-6hs7d\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.138023 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-registry-certificates\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.138023 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.137969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-trusted-ca\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.138023 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.138010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-bound-sa-token\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.143040 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.142995    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:28.239135 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-installation-pull-secrets\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239135 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239118    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-registry-tls\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239135 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-ca-trust-extracted\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239157    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-image-registry-private-configuration\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239292    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hs7d\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-kube-api-access-6hs7d\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-registry-certificates\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239486 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-trusted-ca\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239486 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-bound-sa-token\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.239588 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.239538    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-ca-trust-extracted\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.240343 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.240311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-registry-certificates\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.240569 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.240528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-trusted-ca\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.241984 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.241964    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-installation-pull-secrets\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.242114 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.242097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-registry-tls\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.242194 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.242177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-image-registry-private-configuration\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.255536 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.255512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-bound-sa-token\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.256284 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.256258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hs7d\" (UniqueName: \"kubernetes.io/projected/aba4a02e-1d1b-4d7b-b221-a6a8d29dd083-kube-api-access-6hs7d\") pod \"image-registry-7686b6bc78-kcjfj\" (UID: \"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083\") " pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.292659 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.292634    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:28.375133 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.375104    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:28.393290 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.393267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:28.766100 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.766068    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7686b6bc78-kcjfj"]
Sep 11 13:59:28.863169 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.863137    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bqdgz"
Sep 11 13:59:28.900063 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:28.900017    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba4a02e_1d1b_4d7b_b221_a6a8d29dd083.slice/crio-26dc2d3b34938179ef3a8eabd00e6eb0327803541777249ef94302b84faa9940 WatchSource:0}: Error finding container 26dc2d3b34938179ef3a8eabd00e6eb0327803541777249ef94302b84faa9940: Status 404 returned error can't find the container with id 26dc2d3b34938179ef3a8eabd00e6eb0327803541777249ef94302b84faa9940
Sep 11 13:59:28.967168 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:28.967135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj" event={"ID":"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083","Type":"ContainerStarted","Data":"26dc2d3b34938179ef3a8eabd00e6eb0327803541777249ef94302b84faa9940"}
Sep 11 13:59:29.973099 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:29.973058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c8rm7" event={"ID":"fdf104d2-6dd5-4717-ba07-687b56918990","Type":"ContainerStarted","Data":"9fc5d612dcd7a2fb89a955b690f4bc90a40d475ce6b7d0eb0ae7ac0cd2b55fe9"}
Sep 11 13:59:29.974771 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:29.974739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj" event={"ID":"aba4a02e-1d1b-4d7b-b221-a6a8d29dd083","Type":"ContainerStarted","Data":"8a5d35e191ec4af2ffcdb5ece128ff3269ef3d85d56638b18bd432b0d5022b91"}
Sep 11 13:59:29.974916 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:29.974858    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:29.994662 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:29.994616    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c8rm7" podStartSLOduration=33.612463212 podStartE2EDuration="35.994600218s" totalImagesPullingTime="2.382137006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:58:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:29.994505208 +0000 UTC m=+103.958120201" watchObservedRunningTime="2026-09-11 13:59:29.994600218 +0000 UTC m=+103.958215192"
Sep 11 13:59:30.018039 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:30.017982    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj" podStartSLOduration=2.017967979 podStartE2EDuration="2.017967979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:30.017242483 +0000 UTC m=+103.980857456" watchObservedRunningTime="2026-09-11 13:59:30.017967979 +0000 UTC m=+103.981582952"
Sep 11 13:59:30.874831 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:30.874791    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:30.944859 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:30.944831    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:31.507319 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:31.507283    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:31.588495 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:31.588463    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:31.899909 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:31.899834    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:31.967781 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:31.967748    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.068933 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.068903    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.168576 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.168505    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.231280 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.231236    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.368919 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.368886    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.487196 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.487161    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.633979 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.633954    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.785910 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.785814    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.885186 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.885156    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:32.986241 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:32.986216    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:33.650006 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:33.649969    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:33.786042 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:33.786001    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:33.870512 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:33.870480    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:34.327233 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:34.327198    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:34.995411 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:34.995376    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:35.063265 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:35.063229    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:36.859390 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:36.859354    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:36.964497 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:36.964471    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94z9z"
Sep 11 13:59:37.182191 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.182104    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.221482 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.221455    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"]
Sep 11 13:59:37.224959 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.224941    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.228735 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.228712    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 13:59:37.228735 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.228716    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bfptr\""
Sep 11 13:59:37.228864 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.228746    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 13:59:37.229689 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.229672    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 13:59:37.229771 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.229676    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 13:59:37.229771 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.229722    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 13:59:37.258554 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.258535    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"]
Sep 11 13:59:37.269699 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.269677    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-blfsk"]
Sep 11 13:59:37.271708 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.271690    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.273980 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.273962    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 13:59:37.274103 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.273998    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 13:59:37.274221 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.274204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wckx2\""
Sep 11 13:59:37.274427 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.274413    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 13:59:37.293552 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.293533    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.301481 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.301460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/317fe2d5-83c9-445d-9063-7101718fb92c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.301562 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.301506    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzvdf\" (UniqueName: \"kubernetes.io/projected/317fe2d5-83c9-445d-9063-7101718fb92c-kube-api-access-gzvdf\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.301602 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.301568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/317fe2d5-83c9-445d-9063-7101718fb92c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.301637 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.301620    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/317fe2d5-83c9-445d-9063-7101718fb92c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.393413 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.393388    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.402315 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-accelerators-collector-config\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402418 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-textfile\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402418 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-sys\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402418 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzvdf\" (UniqueName: \"kubernetes.io/projected/317fe2d5-83c9-445d-9063-7101718fb92c-kube-api-access-gzvdf\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.402418 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-tls\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-root\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ccaba7-8d7b-494a-b37f-8e79f1432218-metrics-client-ca\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4md\" (UniqueName: \"kubernetes.io/projected/88ccaba7-8d7b-494a-b37f-8e79f1432218-kube-api-access-9z4md\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/317fe2d5-83c9-445d-9063-7101718fb92c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.402581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-wtmp\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402581 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402574    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.402904 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/317fe2d5-83c9-445d-9063-7101718fb92c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.402904 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.402644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/317fe2d5-83c9-445d-9063-7101718fb92c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.403215 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.403198    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/317fe2d5-83c9-445d-9063-7101718fb92c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.404941 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.404923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/317fe2d5-83c9-445d-9063-7101718fb92c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.405061 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.405043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/317fe2d5-83c9-445d-9063-7101718fb92c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.411922 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.411894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzvdf\" (UniqueName: \"kubernetes.io/projected/317fe2d5-83c9-445d-9063-7101718fb92c-kube-api-access-gzvdf\") pod \"openshift-state-metrics-67dcf8c979-mgx5s\" (UID: \"317fe2d5-83c9-445d-9063-7101718fb92c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.492333 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.492305    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.503228 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-accelerators-collector-config\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503323 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-textfile\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503323 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-sys\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503323 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-tls\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-root\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-sys\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:37.503350    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ccaba7-8d7b-494a-b37f-8e79f1432218-metrics-client-ca\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-root\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4md\" (UniqueName: \"kubernetes.io/projected/88ccaba7-8d7b-494a-b37f-8e79f1432218-kube-api-access-9z4md\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-wtmp\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: E0911 13:59:37.503431    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-tls podName:88ccaba7-8d7b-494a-b37f-8e79f1432218 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:38.003413987 +0000 UTC m=+111.967028952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-tls") pod "node-exporter-blfsk" (UID: "88ccaba7-8d7b-494a-b37f-8e79f1432218") : secret "node-exporter-tls" not found
Sep 11 13:59:37.503479 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503927 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-wtmp\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503927 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-textfile\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503927 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503811    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-accelerators-collector-config\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.503927 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.503818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ccaba7-8d7b-494a-b37f-8e79f1432218-metrics-client-ca\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.505482 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.505464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.512104 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.512083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4md\" (UniqueName: \"kubernetes.io/projected/88ccaba7-8d7b-494a-b37f-8e79f1432218-kube-api-access-9z4md\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:37.534074 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.534054    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"
Sep 11 13:59:37.593768 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.593662    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.722737 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.722712    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.803698 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.803680    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s"]
Sep 11 13:59:37.805446 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:37.805425    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317fe2d5_83c9_445d_9063_7101718fb92c.slice/crio-3735948ae64af5b458ece8ade8629f7b6acaa768ec0ef0fc6bee8d56a1d4391e WatchSource:0}: Error finding container 3735948ae64af5b458ece8ade8629f7b6acaa768ec0ef0fc6bee8d56a1d4391e: Status 404 returned error can't find the container with id 3735948ae64af5b458ece8ade8629f7b6acaa768ec0ef0fc6bee8d56a1d4391e
Sep 11 13:59:37.808588 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.808562    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:37.926637 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:37.926611    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:38.000853 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.000826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s" event={"ID":"317fe2d5-83c9-445d-9063-7101718fb92c","Type":"ContainerStarted","Data":"1940b79d7f9c63e5d400b25f0ca5340e185c71214670dfa9ed04dc327fc953d0"}
Sep 11 13:59:38.000934 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.000879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s" event={"ID":"317fe2d5-83c9-445d-9063-7101718fb92c","Type":"ContainerStarted","Data":"54fe3ff8b4e487ad8d2ec8c0d89677b653bf558b8723976645f20f57d70c0658"}
Sep 11 13:59:38.000934 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.000899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s" event={"ID":"317fe2d5-83c9-445d-9063-7101718fb92c","Type":"ContainerStarted","Data":"3735948ae64af5b458ece8ade8629f7b6acaa768ec0ef0fc6bee8d56a1d4391e"}
Sep 11 13:59:38.007145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.007121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-tls\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:38.009189 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.009167    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88ccaba7-8d7b-494a-b37f-8e79f1432218-node-exporter-tls\") pod \"node-exporter-blfsk\" (UID: \"88ccaba7-8d7b-494a-b37f-8e79f1432218\") " pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:38.036106 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.036080    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:38.180438 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.180388    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blfsk"
Sep 11 13:59:38.189302 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:38.189281    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ccaba7_8d7b_494a_b37f_8e79f1432218.slice/crio-585ccae85ce671d91a20fcdde69fe2379751aa5db1b0c80d1cf8cfa55247a579 WatchSource:0}: Error finding container 585ccae85ce671d91a20fcdde69fe2379751aa5db1b0c80d1cf8cfa55247a579: Status 404 returned error can't find the container with id 585ccae85ce671d91a20fcdde69fe2379751aa5db1b0c80d1cf8cfa55247a579
Sep 11 13:59:38.255344 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:38.255325    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:39.005313 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:39.005273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blfsk" event={"ID":"88ccaba7-8d7b-494a-b37f-8e79f1432218","Type":"ContainerStarted","Data":"585ccae85ce671d91a20fcdde69fe2379751aa5db1b0c80d1cf8cfa55247a579"}
Sep 11 13:59:40.006789 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.006751    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:40.010722 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.010694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s" event={"ID":"317fe2d5-83c9-445d-9063-7101718fb92c","Type":"ContainerStarted","Data":"11164c40a6d7b52c88d29be2869903cfc3d9271026f70daabc721e1b92de7876"}
Sep 11 13:59:40.012240 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.012216    2558 generic.go:356] "Generic (PLEG): container finished" podID="88ccaba7-8d7b-494a-b37f-8e79f1432218" containerID="40ab640d6334eead3f24c5c5dd6b275737a4731ec07bad823350f58ac027f9ac" exitCode=0
Sep 11 13:59:40.012344 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.012270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blfsk" event={"ID":"88ccaba7-8d7b-494a-b37f-8e79f1432218","Type":"ContainerDied","Data":"40ab640d6334eead3f24c5c5dd6b275737a4731ec07bad823350f58ac027f9ac"}
Sep 11 13:59:40.041420 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.041378    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mgx5s" podStartSLOduration=1.792133324 podStartE2EDuration="3.041366093s" totalImagesPullingTime="1.249232769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:40.041190428 +0000 UTC m=+114.004805401" watchObservedRunningTime="2026-09-11 13:59:40.041366093 +0000 UTC m=+114.004981066"
Sep 11 13:59:40.105355 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.105326    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:40.206222 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.206189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:40.595123 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:40.595086    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:41.018468 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.018436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blfsk" event={"ID":"88ccaba7-8d7b-494a-b37f-8e79f1432218","Type":"ContainerStarted","Data":"ed857b325ce62dc947f326dd2b19bc9badfea59b296810668ec7209286d29de2"}
Sep 11 13:59:41.018804 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.018491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blfsk" event={"ID":"88ccaba7-8d7b-494a-b37f-8e79f1432218","Type":"ContainerStarted","Data":"2ed905d881a5806e8496e7f6c0e007e6e7a2e07d3aba4f7a707a57a11b4dada4"}
Sep 11 13:59:41.042141 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.042095    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-blfsk" podStartSLOduration=3.00910245 podStartE2EDuration="4.042080958s" totalImagesPullingTime="969.164753ms" totalInitContainerRuntime="63.813755ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:41.040415107 +0000 UTC m=+115.004030118" watchObservedRunningTime="2026-09-11 13:59:41.042080958 +0000 UTC m=+115.005695934"
Sep 11 13:59:41.574584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.574525    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"]
Sep 11 13:59:41.576470 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.576454    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.580824 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.580795    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 13:59:41.581058 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.581022    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jkvdc\""
Sep 11 13:59:41.583523 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.583502    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 13:59:41.584644 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.584624    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-66ellmn4i4p4l\""
Sep 11 13:59:41.584644 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.584640    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 13:59:41.584781 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.584654    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 13:59:41.598735 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.598713    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"]
Sep 11 13:59:41.733438 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.733415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-secret-metrics-server-client-certs\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.733544 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.733444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-client-ca-bundle\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.733544 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.733461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m547z\" (UniqueName: \"kubernetes.io/projected/b415e54f-a579-444d-85eb-a7c268b85bdf-kube-api-access-m547z\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.733544 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.733524    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-secret-metrics-server-tls\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.733646 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.733569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b415e54f-a579-444d-85eb-a7c268b85bdf-metrics-server-audit-profiles\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.733982 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.733595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b415e54f-a579-444d-85eb-a7c268b85bdf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.734191 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.734172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b415e54f-a579-444d-85eb-a7c268b85bdf-audit-log\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.738832 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.738806    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:41.820186 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.820159    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:41.835089 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b415e54f-a579-444d-85eb-a7c268b85bdf-audit-log\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835089 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-secret-metrics-server-client-certs\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835187 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-client-ca-bundle\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835187 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m547z\" (UniqueName: \"kubernetes.io/projected/b415e54f-a579-444d-85eb-a7c268b85bdf-kube-api-access-m547z\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835187 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-secret-metrics-server-tls\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835326 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b415e54f-a579-444d-85eb-a7c268b85bdf-metrics-server-audit-profiles\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835326 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b415e54f-a579-444d-85eb-a7c268b85bdf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.835444 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.835424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b415e54f-a579-444d-85eb-a7c268b85bdf-audit-log\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.836465 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.836439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b415e54f-a579-444d-85eb-a7c268b85bdf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.836596 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.836576    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b415e54f-a579-444d-85eb-a7c268b85bdf-metrics-server-audit-profiles\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.837941 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.837912    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-client-ca-bundle\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.838053 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.838015    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-secret-metrics-server-client-certs\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.838141 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.838120    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b415e54f-a579-444d-85eb-a7c268b85bdf-secret-metrics-server-tls\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.852856 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.852832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m547z\" (UniqueName: \"kubernetes.io/projected/b415e54f-a579-444d-85eb-a7c268b85bdf-kube-api-access-m547z\") pod \"metrics-server-7fcf85d9b5-rfwgf\" (UID: \"b415e54f-a579-444d-85eb-a7c268b85bdf\") " pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.885833 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.885803    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 13:59:41.902104 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:41.902071    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:42.020939 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:42.020904    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:42.097336 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:42.097274    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:42.159177 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:42.159154    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"]
Sep 11 13:59:42.162205 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:42.162180    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb415e54f_a579_444d_85eb_a7c268b85bdf.slice/crio-c5c1bd62b334e0a73f54b3a2f2f18d00cef280672f2d52834dbc1e9f1a161ef1 WatchSource:0}: Error finding container c5c1bd62b334e0a73f54b3a2f2f18d00cef280672f2d52834dbc1e9f1a161ef1: Status 404 returned error can't find the container with id c5c1bd62b334e0a73f54b3a2f2f18d00cef280672f2d52834dbc1e9f1a161ef1
Sep 11 13:59:43.027931 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.027899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf" event={"ID":"b415e54f-a579-444d-85eb-a7c268b85bdf","Type":"ContainerStarted","Data":"c5c1bd62b334e0a73f54b3a2f2f18d00cef280672f2d52834dbc1e9f1a161ef1"}
Sep 11 13:59:43.471424 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.471394    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:59:43.474431 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.474402    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.484264 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.484225    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 13:59:43.485119 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.485095    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 13:59:43.486051 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.486018    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 13:59:43.486920 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.486898    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 13:59:43.487056 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.486940    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 13:59:43.487893 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.487872    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 13:59:43.492596 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.492574    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 13:59:43.500236 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.500217    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3orjac32ks8iu\""
Sep 11 13:59:43.500442 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.500403    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 13:59:43.500442 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.500433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 13:59:43.500629 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.500456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 13:59:43.500629 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.500433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 13:59:43.500629 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.500557    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ddwqx\""
Sep 11 13:59:43.501583 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.501568    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 13:59:43.503551 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.503533    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 13:59:43.516140 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.516112    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:43.517422 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.517406    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 13:59:43.543066 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.543022    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:59:43.547540 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547635 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547635 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547575    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547635 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547768 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547768 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547768 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-web-config\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547768 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547791    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.547911 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547903    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ngv2\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-kube-api-access-7ngv2\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-config\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.547968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.548006    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.548051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.548075    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548145 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.548107    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-config-out\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.548350 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.548151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.648691 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.648812 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.648812 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.648812 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.648812 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.648812 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649086 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.648825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649209 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-web-config\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649316 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649316 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649427 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649317    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649427 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649427 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649427 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649407    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ngv2\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-kube-api-access-7ngv2\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649623 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-config\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649641    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649682 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649667    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649837 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649837 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649837 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.649837 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.649808    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-config-out\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.650534 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.650509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.651353 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.651058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.651353 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.650510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.651733 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.651704    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.652401 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.652379    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.652663 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.652638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.653162 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.653139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.653715 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.653669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.654342 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.654312    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.654425 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.654377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.654476 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.654422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.654516 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.654477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-config-out\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.655018 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.654994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.655153 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.655133    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-web-config\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.655312 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.655290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.655842 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.655826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.656932 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.656916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-config\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.663761 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.663740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ngv2\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-kube-api-access-7ngv2\") pod \"prometheus-k8s-0\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:43.786818 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:43.786766    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:44.032695 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:44.032660    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf" event={"ID":"b415e54f-a579-444d-85eb-a7c268b85bdf","Type":"ContainerStarted","Data":"d9058b6d698aca286c63208ba644227cdeb6eee67a6eb7ab83d5078504df8458"}
Sep 11 13:59:44.056612 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:44.056533    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf" podStartSLOduration=1.6844413089999999 podStartE2EDuration="3.05651984s" totalImagesPullingTime="1.372078531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:44.054990761 +0000 UTC m=+118.018605746" watchObservedRunningTime="2026-09-11 13:59:44.05651984 +0000 UTC m=+118.020134814"
Sep 11 13:59:44.075381 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:44.075352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:59:44.079630 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:44.079600    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e69fe6_4760_4371_8bbf_693badebc384.slice/crio-7056c7747fe5992d26b135cdfebc515e75bafe6e782088c3314d73367ad7d525 WatchSource:0}: Error finding container 7056c7747fe5992d26b135cdfebc515e75bafe6e782088c3314d73367ad7d525: Status 404 returned error can't find the container with id 7056c7747fe5992d26b135cdfebc515e75bafe6e782088c3314d73367ad7d525
Sep 11 13:59:45.037527 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:45.037493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"7056c7747fe5992d26b135cdfebc515e75bafe6e782088c3314d73367ad7d525"}
Sep 11 13:59:46.042699 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.042662    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da" exitCode=0
Sep 11 13:59:46.043204 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.042754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"}
Sep 11 13:59:46.096283 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.096255    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:46.164181 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.164159    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:46.599998 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.599957    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:46.660512 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.660478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:46.969657 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:46.969619    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:47.090148 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:47.090073    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:50.060442 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:50.060410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"}
Sep 11 13:59:50.060775 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:50.060471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"}
Sep 11 13:59:50.199303 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:50.199270    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:50.982522 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:50.982500    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7686b6bc78-kcjfj"
Sep 11 13:59:51.080239 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:51.080195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"}
Sep 11 13:59:51.080584 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:51.080280    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"}
Sep 11 13:59:51.275527 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:51.275496    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:51.432427 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:51.432366    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:52.087559 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:52.087527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"}
Sep 11 13:59:52.087916 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:52.087604    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerStarted","Data":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"}
Sep 11 13:59:52.144199 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:52.144149    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.404789223 podStartE2EDuration="9.144132471s" totalImagesPullingTime="5.719377905s" totalInitContainerRuntime="19.965343ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:52.14298207 +0000 UTC m=+126.106597080" watchObservedRunningTime="2026-09-11 13:59:52.144132471 +0000 UTC m=+126.107747444"
Sep 11 13:59:52.194263 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:52.194242    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:52.301990 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:52.301964    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:53.787137 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:53.787104    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:59:54.988676 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:54.988640    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:55.102813 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:55.102781    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:55.342447 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:55.342361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:59:55.345020 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:55.344998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd187b1d-bbab-45a2-88a3-1d32dec00e36-metrics-certs\") pod \"network-metrics-daemon-th28r\" (UID: \"dd187b1d-bbab-45a2-88a3-1d32dec00e36\") " pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:59:55.420008 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:55.419973    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh858\""
Sep 11 13:59:55.427354 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:55.427319    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th28r"
Sep 11 13:59:55.713867 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:55.713818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-th28r"]
Sep 11 13:59:55.717999 ip-10-0-140-144 kubenswrapper[2558]: W0911 13:59:55.717971    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd187b1d_bbab_45a2_88a3_1d32dec00e36.slice/crio-a40fde48518ae7802689120d9b8f834bf2fa301d826c65b6f1d1658399e5cf44 WatchSource:0}: Error finding container a40fde48518ae7802689120d9b8f834bf2fa301d826c65b6f1d1658399e5cf44: Status 404 returned error can't find the container with id a40fde48518ae7802689120d9b8f834bf2fa301d826c65b6f1d1658399e5cf44
Sep 11 13:59:56.056040 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:56.055951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:56.104358 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:56.104319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-th28r" event={"ID":"dd187b1d-bbab-45a2-88a3-1d32dec00e36","Type":"ContainerStarted","Data":"a40fde48518ae7802689120d9b8f834bf2fa301d826c65b6f1d1658399e5cf44"}
Sep 11 13:59:56.116801 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:56.116771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:57.110180 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:57.110147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-th28r" event={"ID":"dd187b1d-bbab-45a2-88a3-1d32dec00e36","Type":"ContainerStarted","Data":"f7f4a731076d9cae4b47330dd6e9ace4ff5275a058a645690fb78e8bcaac8956"}
Sep 11 13:59:57.110530 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:57.110198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-th28r" event={"ID":"dd187b1d-bbab-45a2-88a3-1d32dec00e36","Type":"ContainerStarted","Data":"f47098fb4b9343f8fefc7fe2331919e516c05dec6808053ae7c80650c06197ed"}
Sep 11 13:59:57.130938 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:57.130891    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-th28r" podStartSLOduration=129.270220337 podStartE2EDuration="2m10.13087613s" totalImagesPullingTime="860.655793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:57.129505429 +0000 UTC m=+131.093120402" watchObservedRunningTime="2026-09-11 13:59:57.13087613 +0000 UTC m=+131.094491104"
Sep 11 13:59:57.265638 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:57.265610    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:57.354565 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:57.354540    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:58.801372 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:58.801339    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 13:59:58.901690 ip-10-0-140-144 kubenswrapper[2558]: I0911 13:59:58.901654    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:01.002412 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:01.002380    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:01.102989 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:01.102960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:01.886852 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:01.886813    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 14:00:01.887005 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:01.886876    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 14:00:01.942651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:01.942623    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:02.089390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:02.089351    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:06.022943 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:06.022906    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:06.120942 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:06.120907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:06.198699 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:06.198674    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:12.321353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:12.321318    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:12.420966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:12.420928    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:21.891765 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:21.891736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 14:00:21.895851 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:21.895827    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fcf85d9b5-rfwgf"
Sep 11 14:00:26.714244 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:26.714200    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:26.789280 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:26.789241    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:43.787511 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:43.787426    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:00:43.802283 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:43.802260    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:00:44.312079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:44.312022    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:00:56.399882 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:56.399847    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:00:56.703422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:00:56.703385    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:01:01.780086 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780048    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:01:01.780624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780491    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="prometheus" containerID="cri-o://3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035" gracePeriod=600
Sep 11 14:01:01.780624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780531    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="config-reloader" containerID="cri-o://382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6" gracePeriod=600
Sep 11 14:01:01.780624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780551    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy" containerID="cri-o://d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17" gracePeriod=600
Sep 11 14:01:01.780624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780524    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="thanos-sidecar" containerID="cri-o://f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b" gracePeriod=600
Sep 11 14:01:01.780624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780496    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-thanos" containerID="cri-o://430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a" gracePeriod=600
Sep 11 14:01:01.780935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:01.780506    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-web" containerID="cri-o://4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9" gracePeriod=600
Sep 11 14:01:02.025498 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.025468    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.147175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147092    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-config\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147139    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-metrics-client-ca\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147168    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-kubelet-serving-ca-bundle\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-kube-rbac-proxy\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147232    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-0\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147265    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ngv2\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-kube-api-access-7ngv2\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-2\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147319    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147378    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-tls-assets\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147547    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.147740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147602    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.147740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147654    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-1\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147724    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-thanos-prometheus-http-client-file\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-config-out\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147796    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-grpc-tls\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147820    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-trusted-ca-bundle\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147854    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-tls\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147877    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-serving-certs-ca-bundle\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147927    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-metrics-client-certs\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147960    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-web-config\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.147991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.147989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-db\") pod \"d1e69fe6-4760-4371-8bbf-693badebc384\" (UID: \"d1e69fe6-4760-4371-8bbf-693badebc384\") "
Sep 11 14:01:02.148416 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.148389    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.149723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.148702    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.149723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.149589    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.149723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.149611    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-metrics-client-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.149723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.149626    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.149723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.149641    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.150089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.149854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.150089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.149950    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.150595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150361    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.150595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150416    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:01:02.150595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150533    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:01:02.150930 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150897    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.150930 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-config" (OuterVolumeSpecName: "config") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.151079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:01:02.151079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.150981    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-kube-api-access-7ngv2" (OuterVolumeSpecName: "kube-api-access-7ngv2") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "kube-api-access-7ngv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:01:02.151175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.151111    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.152294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.152266    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.152589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.152565    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.152687 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.152601    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.152687 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.152653    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.152687 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.152664    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-config-out" (OuterVolumeSpecName: "config-out") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:01:02.161557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.161533    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-web-config" (OuterVolumeSpecName: "web-config") pod "d1e69fe6-4760-4371-8bbf-693badebc384" (UID: "d1e69fe6-4760-4371-8bbf-693badebc384"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:02.249991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.249954    2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-config\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.249991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.249990    2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-kube-rbac-proxy\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.249991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250001    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ngv2\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-kube-api-access-7ngv2\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250011    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250021    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1e69fe6-4760-4371-8bbf-693badebc384-tls-assets\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250048    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250056    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250065    2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250073    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-config-out\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250081    2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-grpc-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250088    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250098    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250106    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1e69fe6-4760-4371-8bbf-693badebc384-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250114    2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-secret-metrics-client-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250122    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1e69fe6-4760-4371-8bbf-693badebc384-web-config\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.250186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.250130    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1e69fe6-4760-4371-8bbf-693badebc384-prometheus-k8s-db\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:01:02.364635 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364604    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a" exitCode=0
Sep 11 14:01:02.364635 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364630    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17" exitCode=0
Sep 11 14:01:02.364635 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364636    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9" exitCode=0
Sep 11 14:01:02.364635 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364643    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b" exitCode=0
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364648    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6" exitCode=0
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364656    2558 generic.go:356] "Generic (PLEG): container finished" podID="d1e69fe6-4760-4371-8bbf-693badebc384" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035" exitCode=0
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364678    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"}
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"}
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"}
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364793    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"}
Sep 11 14:01:02.364873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364815    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.365141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"}
Sep 11 14:01:02.365141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364927    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"}
Sep 11 14:01:02.365141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.364951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1e69fe6-4760-4371-8bbf-693badebc384","Type":"ContainerDied","Data":"7056c7747fe5992d26b135cdfebc515e75bafe6e782088c3314d73367ad7d525"}
Sep 11 14:01:02.373366 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.373348    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.379798 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.379783    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.386760 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.386742    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.390419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.390396    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:01:02.393893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.393871    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.396163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.396142    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:01:02.402342 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.402327    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.408940 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.408924    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.414986 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.414973    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.415311 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.415288    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.415362 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.415323    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"} err="failed to get container status \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist"
Sep 11 14:01:02.415362 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.415355    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.415615 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.415598    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.415657 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.415620    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"} err="failed to get container status \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist"
Sep 11 14:01:02.415657 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.415637    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.415830 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.415816    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.415873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.415833    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"} err="failed to get container status \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist"
Sep 11 14:01:02.415873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.415844    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.416061 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.416044    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.416106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416066    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"} err="failed to get container status \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist"
Sep 11 14:01:02.416106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416079    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.416288 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.416272    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.416326 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416292    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"} err="failed to get container status \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist"
Sep 11 14:01:02.416326 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416306    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.416501 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.416487    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.416540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416505    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"} err="failed to get container status \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist"
Sep 11 14:01:02.416540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416515    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.416696 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:01:02.416682    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.416732 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416698    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"} err="failed to get container status \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist"
Sep 11 14:01:02.416732 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416709    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.416893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416877    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"} err="failed to get container status \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist"
Sep 11 14:01:02.416934 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.416893    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.417089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417073    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"} err="failed to get container status \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist"
Sep 11 14:01:02.417089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417088    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.417319 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417300    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"} err="failed to get container status \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist"
Sep 11 14:01:02.417368 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417320    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.417504 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417489    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"} err="failed to get container status \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist"
Sep 11 14:01:02.417552 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417503    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.417703 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417688    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"} err="failed to get container status \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist"
Sep 11 14:01:02.417747 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417704    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.417883 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417868    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"} err="failed to get container status \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist"
Sep 11 14:01:02.417936 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.417884    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.418156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418139    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"} err="failed to get container status \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist"
Sep 11 14:01:02.418218 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418158    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.418353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418337    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"} err="failed to get container status \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist"
Sep 11 14:01:02.418398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418353    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.418544 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418530    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"} err="failed to get container status \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist"
Sep 11 14:01:02.418588 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418544    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.418745 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418730    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"} err="failed to get container status \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist"
Sep 11 14:01:02.418785 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418746    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.418928 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418913    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"} err="failed to get container status \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist"
Sep 11 14:01:02.418972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.418928    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.419163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419146    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"} err="failed to get container status \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist"
Sep 11 14:01:02.419219 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419164    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.419342 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419326    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"} err="failed to get container status \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist"
Sep 11 14:01:02.419386 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419342    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.419524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419509    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"} err="failed to get container status \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist"
Sep 11 14:01:02.419571 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419524    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.419726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419710    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"} err="failed to get container status \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist"
Sep 11 14:01:02.419767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419728    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.419952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419937    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"} err="failed to get container status \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist"
Sep 11 14:01:02.419999 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.419953    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.420173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420157    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"} err="failed to get container status \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist"
Sep 11 14:01:02.420217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420173    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.420378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420363    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"} err="failed to get container status \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist"
Sep 11 14:01:02.420423 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420377    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.420592 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420571    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"} err="failed to get container status \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist"
Sep 11 14:01:02.420592 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420587    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.420817 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420798    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"} err="failed to get container status \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist"
Sep 11 14:01:02.420817 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.420817    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.421041 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421010    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"} err="failed to get container status \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist"
Sep 11 14:01:02.421041 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421038    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.421250 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421232    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"} err="failed to get container status \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist"
Sep 11 14:01:02.421291 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421251    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.421448 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421429    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"} err="failed to get container status \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist"
Sep 11 14:01:02.421488 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421449    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.421677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421658    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"} err="failed to get container status \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist"
Sep 11 14:01:02.421723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421678    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.421993 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421909    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"} err="failed to get container status \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist"
Sep 11 14:01:02.421993 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.421940    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.422213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422196    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"} err="failed to get container status \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist"
Sep 11 14:01:02.422260 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422214    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.422432 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422414    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"} err="failed to get container status \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist"
Sep 11 14:01:02.422470 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422432    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.422622 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422607    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"} err="failed to get container status \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist"
Sep 11 14:01:02.422658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422622    2558 scope.go:117] "RemoveContainer" containerID="430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"
Sep 11 14:01:02.422826 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422808    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a"} err="failed to get container status \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": rpc error: code = NotFound desc = could not find container \"430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a\": container with ID starting with 430d6eee0d1b4be7e845d013ef66f60cb87d9f8d63d783533cc875aee1c2fd2a not found: ID does not exist"
Sep 11 14:01:02.422864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.422828    2558 scope.go:117] "RemoveContainer" containerID="d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"
Sep 11 14:01:02.423021 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423006    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17"} err="failed to get container status \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": rpc error: code = NotFound desc = could not find container \"d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17\": container with ID starting with d90e7233b209a3d383d7120d4a435f5c1c68094a921ca3fb4156bb11e6e0da17 not found: ID does not exist"
Sep 11 14:01:02.423085 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423021    2558 scope.go:117] "RemoveContainer" containerID="4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"
Sep 11 14:01:02.423258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423241    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9"} err="failed to get container status \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": rpc error: code = NotFound desc = could not find container \"4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9\": container with ID starting with 4e4c5a9c49d69b0411b0f58b2f3c256e5842233d0bbbc9618bbbc910ac7415f9 not found: ID does not exist"
Sep 11 14:01:02.423258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423257    2558 scope.go:117] "RemoveContainer" containerID="f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"
Sep 11 14:01:02.423430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423414    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b"} err="failed to get container status \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": rpc error: code = NotFound desc = could not find container \"f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b\": container with ID starting with f28ba0b038745a6fa55962c034dd0fa5108d8a1f113cf0a26309c52920c7eb2b not found: ID does not exist"
Sep 11 14:01:02.423473 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423431    2558 scope.go:117] "RemoveContainer" containerID="382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"
Sep 11 14:01:02.423628 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423611    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6"} err="failed to get container status \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": rpc error: code = NotFound desc = could not find container \"382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6\": container with ID starting with 382edf8fc4286ca02ab729884577dce69bbf4e527c481f1a4beb457fd9183ac6 not found: ID does not exist"
Sep 11 14:01:02.423667 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423629    2558 scope.go:117] "RemoveContainer" containerID="3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"
Sep 11 14:01:02.423817 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423802    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035"} err="failed to get container status \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": rpc error: code = NotFound desc = could not find container \"3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035\": container with ID starting with 3817977321d48b357bf381eb0547b9fb1320d245ccbb939ec005ab4817a3a035 not found: ID does not exist"
Sep 11 14:01:02.423856 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423817    2558 scope.go:117] "RemoveContainer" containerID="6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"
Sep 11 14:01:02.423995 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.423979    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da"} err="failed to get container status \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": rpc error: code = NotFound desc = could not find container \"6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da\": container with ID starting with 6106ed42588ab3885ac8e49ad594b554fad546913f1cfd13d8a9845592b531da not found: ID does not exist"
Sep 11 14:01:02.434895 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.434876    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:01:02.435165 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435152    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="thanos-sidecar"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435167    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="thanos-sidecar"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435176    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="prometheus"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435181    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="prometheus"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435191    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="init-config-reloader"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435196    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="init-config-reloader"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435205    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="config-reloader"
Sep 11 14:01:02.435213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435210    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="config-reloader"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435216    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-web"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435222    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-web"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435229    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435234    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435239    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-thanos"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435245    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-thanos"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435291    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="config-reloader"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435297    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="thanos-sidecar"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435301    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="prometheus"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435306    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-web"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435310    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy"
Sep 11 14:01:02.435397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.435315    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" containerName="kube-rbac-proxy-thanos"
Sep 11 14:01:02.440320 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.440304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.445352 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.445333    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 14:01:02.445608 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.445590    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 14:01:02.445706 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.445633    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 14:01:02.448209 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448192    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 14:01:02.448322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448309    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 14:01:02.448408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448381    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 14:01:02.448481 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448403    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448777    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3orjac32ks8iu\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448838    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448852    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448868    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448888    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 14:01:02.448957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.448845    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ddwqx\""
Sep 11 14:01:02.454524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.454503    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 14:01:02.454524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.454515    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 14:01:02.462272 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.462219    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:01:02.501859 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.501831    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e69fe6-4760-4371-8bbf-693badebc384" path="/var/lib/kubelet/pods/d1e69fe6-4760-4371-8bbf-693badebc384/volumes"
Sep 11 14:01:02.552326 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552304    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552520 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552458    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552520 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552520 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552499    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552520 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2972\" (UniqueName: \"kubernetes.io/projected/60107236-15aa-40f4-b297-4926b57eec65-kube-api-access-m2972\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552618    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552643    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-config\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60107236-15aa-40f4-b297-4926b57eec65-config-out\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552711    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60107236-15aa-40f4-b297-4926b57eec65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552764    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552787    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-web-config\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.552912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.552814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653377 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653377 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60107236-15aa-40f4-b297-4926b57eec65-config-out\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653377 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60107236-15aa-40f4-b297-4926b57eec65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653406    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653437    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-web-config\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653480    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653516    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.653595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653598    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653627    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2972\" (UniqueName: \"kubernetes.io/projected/60107236-15aa-40f4-b297-4926b57eec65-kube-api-access-m2972\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.653793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-config\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654487 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.654164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654487 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.654291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.654487 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.654387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.656793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.655596    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.656793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.656151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.656793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.656382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60107236-15aa-40f4-b297-4926b57eec65-config-out\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.656793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.656618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-config\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.656793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.656740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60107236-15aa-40f4-b297-4926b57eec65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.656793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.656744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.657257 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.656924    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.657257 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.657112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.657502 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.657467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.657969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.657937    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.658231 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.658209    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-web-config\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.658617 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.658591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.658817 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.658797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.658942 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.658925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.659406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.659387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60107236-15aa-40f4-b297-4926b57eec65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.659587 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.659569    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60107236-15aa-40f4-b297-4926b57eec65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.663435 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.663415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2972\" (UniqueName: \"kubernetes.io/projected/60107236-15aa-40f4-b297-4926b57eec65-kube-api-access-m2972\") pod \"prometheus-k8s-0\" (UID: \"60107236-15aa-40f4-b297-4926b57eec65\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:02.749910 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:02.749886    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:03.027129 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:03.027078    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 14:01:03.029734 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:01:03.029706    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60107236_15aa_40f4_b297_4926b57eec65.slice/crio-aa8a01de77af806639e7ec21f5d13f632719459743f55a3a50d01e761cff4831 WatchSource:0}: Error finding container aa8a01de77af806639e7ec21f5d13f632719459743f55a3a50d01e761cff4831: Status 404 returned error can't find the container with id aa8a01de77af806639e7ec21f5d13f632719459743f55a3a50d01e761cff4831
Sep 11 14:01:03.374353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:03.374273    2558 generic.go:356] "Generic (PLEG): container finished" podID="60107236-15aa-40f4-b297-4926b57eec65" containerID="5182c926247ba1a12e8846b88fc9312b9c0b8f561ebd36982707273d3af3651f" exitCode=0
Sep 11 14:01:03.374353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:03.374343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerDied","Data":"5182c926247ba1a12e8846b88fc9312b9c0b8f561ebd36982707273d3af3651f"}
Sep 11 14:01:03.374527 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:03.374418    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"aa8a01de77af806639e7ec21f5d13f632719459743f55a3a50d01e761cff4831"}
Sep 11 14:01:04.381528 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.381491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"9290b5ddcb254be1a5b24af76c0e2f334bee7dcb965687f4acc3926078165aeb"}
Sep 11 14:01:04.381970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.381561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"d17b2b3fd3a5c4a2d0c90c220b990a43cf391b855566b8a3da32543318df2dc7"}
Sep 11 14:01:04.381970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.381589    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"e693b523a6e5265fd53e1fa5354f071f172c416d56534dda508f6c2e51843e0e"}
Sep 11 14:01:04.381970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.381613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"b30e90a031a2e2eaf1d7df186db38fb2d824ab9bf515e5562cda5ad8465d9300"}
Sep 11 14:01:04.381970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.381635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"f39d58e1f964b1561d8697c905ba59af0f5ecd46237eb20ff5da9febb77a4dc4"}
Sep 11 14:01:04.381970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.381669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60107236-15aa-40f4-b297-4926b57eec65","Type":"ContainerStarted","Data":"66c9cf171532d9f240717ed45cc762831a8652d90520db58f6939982cce8892e"}
Sep 11 14:01:04.470533 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.470485    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.451649834 podStartE2EDuration="2.470470544s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.82071ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:04.46670684 +0000 UTC m=+198.430321813" watchObservedRunningTime="2026-09-11 14:01:04.470470544 +0000 UTC m=+198.434085517"
Sep 11 14:01:04.513777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.513744    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:01:04.609288 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:04.609262    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:01:07.750953 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:07.750920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:01:15.397077 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:15.397019    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:01:15.498702 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:15.498670    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:01:15.565612 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:15.565583    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:01:39.752600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:01:39.752567    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:02.750330 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:02.750298    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:02:02.764290 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:02.764265    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:02:03.599658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:03.599632    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 14:02:11.668343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:11.668236    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:11.769983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:11.769942    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:16.525406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:16.525377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:16.625373 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:16.625339    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:16.691155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:16.691121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:17.301613 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:17.301580    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:17.400178 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:17.400114    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:23.665738 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:23.665694    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:23.815353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:23.815323    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:26.737533 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:26.737496    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:26.832595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:26.832569    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:26.933133 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:26.933105    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:27.541481 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:27.541447    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:27.642902 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:27.642872    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:27.732474 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:27.732448    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:30.559680 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:30.559642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:30.710781 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:30.710741    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:46.420737 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:46.420709    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:02:46.422211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:46.422190    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:02:46.426852 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:46.426831    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 14:02:48.332894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:48.332861    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:02:48.476581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:02:48.476546    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:03:01.813626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:03:01.813581    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:03:10.245710 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:03:10.245669    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:03:10.338138 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:03:10.338104    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:03:22.526685 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:03:22.526649    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:04:30.390521 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:04:30.390484    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:04:49.402649 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:04:49.402614    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:04:49.515244 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:04:49.515211    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:04:58.413719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:04:58.413688    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:04:58.847192 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:04:58.847148    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:00.145005 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:00.144962    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:00.251987 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:00.251952    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:00.395969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:00.395897    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:00.509977 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:00.509940    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:02.213524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:02.213493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:02.410704 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:02.410670    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:03.417660 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:03.417633    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:03.564750 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:03.564721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:04.519160 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:04.519123    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:04.622483 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:04.622436    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:06.969475 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:06.969428    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:07.147275 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:07.147245    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:08.817593 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:08.817564    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:08.919266 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:08.919196    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:14.565372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:14.565338    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:19.613952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:19.613919    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:19.714287 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:19.714249    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:28.925198 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:28.925160    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:29.030431 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:29.030399    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:31.179043 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:31.179002    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:38.175445 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:38.175412    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:38.245042 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:38.245000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:43.510504 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.510472    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:43.586340 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.586306    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-779774756-jc4c7"]
Sep 11 14:05:43.592827 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.592800    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.596566 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.596542    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 14:05:43.596780 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.596547    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 14:05:43.596868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.596548    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 14:05:43.596868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.596567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:05:43.596868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.596565    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jgfjv\""
Sep 11 14:05:43.597197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.596612    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 14:05:43.604725 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.604701    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-779774756-jc4c7"]
Sep 11 14:05:43.614750 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.614728    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:43.706970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.706933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b52c9317-49c4-4dc0-80b9-5437ed074ed2-manager-config\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.707180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.706977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76qv\" (UniqueName: \"kubernetes.io/projected/b52c9317-49c4-4dc0-80b9-5437ed074ed2-kube-api-access-p76qv\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.707180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.707062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b52c9317-49c4-4dc0-80b9-5437ed074ed2-metrics-cert\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.707180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.707112    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b52c9317-49c4-4dc0-80b9-5437ed074ed2-cert\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.807730 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.807645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b52c9317-49c4-4dc0-80b9-5437ed074ed2-metrics-cert\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.807730 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.807693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b52c9317-49c4-4dc0-80b9-5437ed074ed2-cert\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.807946 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.807814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b52c9317-49c4-4dc0-80b9-5437ed074ed2-manager-config\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.807946 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.807857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76qv\" (UniqueName: \"kubernetes.io/projected/b52c9317-49c4-4dc0-80b9-5437ed074ed2-kube-api-access-p76qv\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.808514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.808493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b52c9317-49c4-4dc0-80b9-5437ed074ed2-manager-config\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.810092 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.810071    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b52c9317-49c4-4dc0-80b9-5437ed074ed2-cert\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.810201 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.810164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b52c9317-49c4-4dc0-80b9-5437ed074ed2-metrics-cert\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.820853 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.820829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76qv\" (UniqueName: \"kubernetes.io/projected/b52c9317-49c4-4dc0-80b9-5437ed074ed2-kube-api-access-p76qv\") pod \"lws-controller-manager-779774756-jc4c7\" (UID: \"b52c9317-49c4-4dc0-80b9-5437ed074ed2\") " pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:43.903244 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:43.903210    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:44.188335 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:44.188309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-779774756-jc4c7"]
Sep 11 14:05:44.190670 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:05:44.190646    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52c9317_49c4_4dc0_80b9_5437ed074ed2.slice/crio-3c7d5b5d092ee9463f496e5b4a92136cef29c5c2284ddb8628a0e31463c67d4f WatchSource:0}: Error finding container 3c7d5b5d092ee9463f496e5b4a92136cef29c5c2284ddb8628a0e31463c67d4f: Status 404 returned error can't find the container with id 3c7d5b5d092ee9463f496e5b4a92136cef29c5c2284ddb8628a0e31463c67d4f
Sep 11 14:05:44.192451 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:44.192435    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:05:44.368040 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:44.367995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7" event={"ID":"b52c9317-49c4-4dc0-80b9-5437ed074ed2","Type":"ContainerStarted","Data":"3c7d5b5d092ee9463f496e5b4a92136cef29c5c2284ddb8628a0e31463c67d4f"}
Sep 11 14:05:45.702379 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:45.702340    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:45.830143 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:45.830111    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:46.330866 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.330833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:46.655727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.655638    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:46.762502 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.762418    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:46.823008 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.822974    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"]
Sep 11 14:05:46.826426 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.826403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:46.829184 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.829161    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 14:05:46.829322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.829229    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2hc6w\""
Sep 11 14:05:46.830079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.829944    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 14:05:46.830201 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.830145    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 14:05:46.830268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.830241    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 14:05:46.856803 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.856766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"]
Sep 11 14:05:46.933969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.933886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4f5680e-99e4-4e38-9fc2-72f4c674464e-tmp\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:46.933969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.933929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcd92\" (UniqueName: \"kubernetes.io/projected/f4f5680e-99e4-4e38-9fc2-72f4c674464e-kube-api-access-gcd92\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:46.934196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.934023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f4f5680e-99e4-4e38-9fc2-72f4c674464e-manifests\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:46.934196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.934107    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4f5680e-99e4-4e38-9fc2-72f4c674464e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:46.934196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.934141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f4f5680e-99e4-4e38-9fc2-72f4c674464e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:46.991645 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:46.991611    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:47.034838 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.034803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4f5680e-99e4-4e38-9fc2-72f4c674464e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.035013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.034865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f4f5680e-99e4-4e38-9fc2-72f4c674464e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.035013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.034938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4f5680e-99e4-4e38-9fc2-72f4c674464e-tmp\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.035013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.034964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcd92\" (UniqueName: \"kubernetes.io/projected/f4f5680e-99e4-4e38-9fc2-72f4c674464e-kube-api-access-gcd92\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.035013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.035002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f4f5680e-99e4-4e38-9fc2-72f4c674464e-manifests\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.035581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.035455    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f4f5680e-99e4-4e38-9fc2-72f4c674464e-manifests\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.039058 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.038590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f4f5680e-99e4-4e38-9fc2-72f4c674464e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.039058 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.038798    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4f5680e-99e4-4e38-9fc2-72f4c674464e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.043570 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.043519    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4f5680e-99e4-4e38-9fc2-72f4c674464e-tmp\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.062165 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.062142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcd92\" (UniqueName: \"kubernetes.io/projected/f4f5680e-99e4-4e38-9fc2-72f4c674464e-kube-api-access-gcd92\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-ffqqg\" (UID: \"f4f5680e-99e4-4e38-9fc2-72f4c674464e\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.121990 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.121949    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:47.141124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.141081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:47.268169 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.268137    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:47.451881 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.451843    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"]
Sep 11 14:05:47.500400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.500351    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:47.608848 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.608798    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:47.717955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.717840    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:47.806214 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:47.806184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:48.371619 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:05:48.371583    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f5680e_99e4_4e38_9fc2_72f4c674464e.slice/crio-d6d1a239b5a000ab27027c1e09fdecd60ed80fdc4e00cee41f1b66681ae7ca78 WatchSource:0}: Error finding container d6d1a239b5a000ab27027c1e09fdecd60ed80fdc4e00cee41f1b66681ae7ca78: Status 404 returned error can't find the container with id d6d1a239b5a000ab27027c1e09fdecd60ed80fdc4e00cee41f1b66681ae7ca78
Sep 11 14:05:48.383429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:48.383402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg" event={"ID":"f4f5680e-99e4-4e38-9fc2-72f4c674464e","Type":"ContainerStarted","Data":"d6d1a239b5a000ab27027c1e09fdecd60ed80fdc4e00cee41f1b66681ae7ca78"}
Sep 11 14:05:49.390955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:49.390734    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7" event={"ID":"b52c9317-49c4-4dc0-80b9-5437ed074ed2","Type":"ContainerStarted","Data":"9c0f92c2e28846319e3c92c4f3065d731e29fd0ff8f4a626c48937a760fd9283"}
Sep 11 14:05:49.390955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:49.390814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:05:49.409210 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:49.409158    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7" podStartSLOduration=2.165915653 podStartE2EDuration="6.409136372s" totalImagesPullingTime="4.243220719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:49.408270236 +0000 UTC m=+483.371885228" watchObservedRunningTime="2026-09-11 14:05:49.409136372 +0000 UTC m=+483.372751348"
Sep 11 14:05:51.401450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:51.401413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg" event={"ID":"f4f5680e-99e4-4e38-9fc2-72f4c674464e","Type":"ContainerStarted","Data":"44e074f4ebc9fb488606ea75411b4fff944e46386341e184587dc2e7699e93ea"}
Sep 11 14:05:52.409218 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:52.409180    2558 generic.go:356] "Generic (PLEG): container finished" podID="f4f5680e-99e4-4e38-9fc2-72f4c674464e" containerID="44e074f4ebc9fb488606ea75411b4fff944e46386341e184587dc2e7699e93ea" exitCode=0
Sep 11 14:05:52.409609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:52.409320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg" event={"ID":"f4f5680e-99e4-4e38-9fc2-72f4c674464e","Type":"ContainerDied","Data":"44e074f4ebc9fb488606ea75411b4fff944e46386341e184587dc2e7699e93ea"}
Sep 11 14:05:53.414452 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:53.414373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg" event={"ID":"f4f5680e-99e4-4e38-9fc2-72f4c674464e","Type":"ContainerStarted","Data":"a2c39623e2fc82ee4af5734618b4602515395d761279a1d1f889a0a255a7a1c4"}
Sep 11 14:05:53.414871 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:53.414508    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:05:53.438673 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:53.438624    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg" podStartSLOduration=2.806419988 podStartE2EDuration="7.438608657s" totalImagesPullingTime="3.570526052s" totalInitContainerRuntime="1.061662617s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:53.436779906 +0000 UTC m=+487.400394882" watchObservedRunningTime="2026-09-11 14:05:53.438608657 +0000 UTC m=+487.402223630"
Sep 11 14:05:59.505537 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:59.505504    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:05:59.572436 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:05:59.572407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:00.398457 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:00.398428    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-779774756-jc4c7"
Sep 11 14:06:04.420189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:04.420159    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-ffqqg"
Sep 11 14:06:06.116209 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:06.116173    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:06.291304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:06.291269    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:07.023230 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.023200    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:07.127153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.127122    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"]
Sep 11 14:06:07.138684 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.138666    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.143987 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.143966    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 14:06:07.145140 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.144874    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 14:06:07.145212 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.145169    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pgxcj\""
Sep 11 14:06:07.153381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.153361    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"]
Sep 11 14:06:07.291565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.291483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq6c\" (UniqueName: \"kubernetes.io/projected/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-kube-api-access-lwq6c\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.291565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.291554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-tmp\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.291709 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.291608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-tls-certs\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.386893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.386863    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:07.392843 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.392812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-tmp\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.392960 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.392856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-tls-certs\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.392960 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.392887    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq6c\" (UniqueName: \"kubernetes.io/projected/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-kube-api-access-lwq6c\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.395106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.395086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-tmp\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.395407 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.395385    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-tls-certs\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.421113 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.421086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq6c\" (UniqueName: \"kubernetes.io/projected/7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e-kube-api-access-lwq6c\") pod \"kube-auth-proxy-8494df7896-ch7vt\" (UID: \"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.447890 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.447860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"
Sep 11 14:06:07.739141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:07.739119    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8494df7896-ch7vt"]
Sep 11 14:06:07.741672 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:06:07.741647    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7804bec9_7ec2_43d4_b7d4_4f9e40a6ed7e.slice/crio-109a1830e30030f27a42365a883c38005bcc3553f35ba11902b1298e0a68ee69 WatchSource:0}: Error finding container 109a1830e30030f27a42365a883c38005bcc3553f35ba11902b1298e0a68ee69: Status 404 returned error can't find the container with id 109a1830e30030f27a42365a883c38005bcc3553f35ba11902b1298e0a68ee69
Sep 11 14:06:08.470134 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:08.470101    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt" event={"ID":"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e","Type":"ContainerStarted","Data":"109a1830e30030f27a42365a883c38005bcc3553f35ba11902b1298e0a68ee69"}
Sep 11 14:06:09.622273 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:09.622225    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:09.765376 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:09.765319    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:11.485312 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:11.485276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt" event={"ID":"7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e","Type":"ContainerStarted","Data":"8bee6ef42dd88c438d0455f6127b198676b1817aeb75b1844b2ef396c6d42658"}
Sep 11 14:06:11.532909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:11.532858    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8494df7896-ch7vt" podStartSLOduration=1.340983501 podStartE2EDuration="4.53284581s" totalImagesPullingTime="3.191862309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:11.531913301 +0000 UTC m=+505.495528274" watchObservedRunningTime="2026-09-11 14:06:11.53284581 +0000 UTC m=+505.496460832"
Sep 11 14:06:12.097290 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:12.097256    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:14.844932 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:14.844895    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:14.950617 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:14.950581    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:15.720848 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:15.720810    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:15.813536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:15.813484    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:15.943792 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:15.943740    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:17.384856 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:17.384823    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:17.503363 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:17.503323    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:17.639465 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:17.639369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:17.915983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:17.915877    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:18.167201 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:18.167115    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:28.923413 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:28.923377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:29.083956 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:29.083923    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:29.387661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:29.387627    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:29.666445 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:29.666365    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:06:29.935279 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:06:29.935195    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:19.044965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:19.044925    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:19.139707 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:19.139673    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:22.145226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:22.145184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:34.310781 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:34.310743    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:35.094842 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:35.094811    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:35.934593 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:35.934560    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:37.998933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:37.998884    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:38.098396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:38.098362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:39.121591 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:39.121559    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:39.393063 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:39.392977    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:39.535707 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:39.535681    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:39.658015 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:39.657945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:42.532950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:42.532914    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:44.176694 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:44.176655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:44.324697 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:44.324663    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:45.982472 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:45.982442    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"]
Sep 11 14:07:45.984691 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:45.984672    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:07:45.987300 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:45.987277    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 14:07:45.987406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:45.987323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fnz9v\""
Sep 11 14:07:45.988832 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:45.988814    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 14:07:45.989150 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:45.989135    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 14:07:46.001931 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.001907    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"]
Sep 11 14:07:46.021782 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.021757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh4n\" (UniqueName: \"kubernetes.io/projected/b454f9b7-f90c-459a-b74b-b8a56d454448-kube-api-access-lhh4n\") pod \"dns-operator-controller-manager-7b84cd7c9c-7vf8d\" (UID: \"b454f9b7-f90c-459a-b74b-b8a56d454448\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:07:46.122417 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.122395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh4n\" (UniqueName: \"kubernetes.io/projected/b454f9b7-f90c-459a-b74b-b8a56d454448-kube-api-access-lhh4n\") pod \"dns-operator-controller-manager-7b84cd7c9c-7vf8d\" (UID: \"b454f9b7-f90c-459a-b74b-b8a56d454448\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:07:46.132547 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.132522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh4n\" (UniqueName: \"kubernetes.io/projected/b454f9b7-f90c-459a-b74b-b8a56d454448-kube-api-access-lhh4n\") pod \"dns-operator-controller-manager-7b84cd7c9c-7vf8d\" (UID: \"b454f9b7-f90c-459a-b74b-b8a56d454448\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:07:46.294552 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.294481    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:07:46.445851 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.445830    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:07:46.446343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.446325    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:07:46.566573 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.566484    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"]
Sep 11 14:07:46.569097 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:07:46.569070    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb454f9b7_f90c_459a_b74b_b8a56d454448.slice/crio-7fa3081e1158596f0d252fde32c5d6253f44dc1f9eed8965121c63d54c9e9d16 WatchSource:0}: Error finding container 7fa3081e1158596f0d252fde32c5d6253f44dc1f9eed8965121c63d54c9e9d16: Status 404 returned error can't find the container with id 7fa3081e1158596f0d252fde32c5d6253f44dc1f9eed8965121c63d54c9e9d16
Sep 11 14:07:46.792408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.792383    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:46.874149 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:46.874088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d" event={"ID":"b454f9b7-f90c-459a-b74b-b8a56d454448","Type":"ContainerStarted","Data":"7fa3081e1158596f0d252fde32c5d6253f44dc1f9eed8965121c63d54c9e9d16"}
Sep 11 14:07:47.060790 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.060763    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"]
Sep 11 14:07:47.063505 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.063490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.065597 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.065575    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:47.066489 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.066473    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 14:07:47.066633 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.066615    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 14:07:47.066725 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.066710    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fj6mk\""
Sep 11 14:07:47.086418 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.086395    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"]
Sep 11 14:07:47.130807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.130757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlcm\" (UniqueName: \"kubernetes.io/projected/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-kube-api-access-kwlcm\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.130807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.130786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.130939 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.130820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.231826 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.231799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.231933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.231862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlcm\" (UniqueName: \"kubernetes.io/projected/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-kube-api-access-kwlcm\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.231933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.231883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.232008 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:07:47.231975    2558 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 14:07:47.232070 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:07:47.232046    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-plugin-serving-cert podName:f6a0a2b5-adde-4408-8f6f-aded4c762bf8 nodeName:}" failed. No retries permitted until 2026-09-11 14:07:47.732011356 +0000 UTC m=+601.695626308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-frt84" (UID: "f6a0a2b5-adde-4408-8f6f-aded4c762bf8") : secret "plugin-serving-cert" not found
Sep 11 14:07:47.232444 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.232420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.268409 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.268384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlcm\" (UniqueName: \"kubernetes.io/projected/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-kube-api-access-kwlcm\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.469465 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.469437    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:47.735915 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.735812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.738985 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.738958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a0a2b5-adde-4408-8f6f-aded4c762bf8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-frt84\" (UID: \"f6a0a2b5-adde-4408-8f6f-aded4c762bf8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:47.987644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:47.987545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"
Sep 11 14:07:48.277524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:48.277494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84"]
Sep 11 14:07:49.140722 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:07:49.140685    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a0a2b5_adde_4408_8f6f_aded4c762bf8.slice/crio-d03007412f277fbe2703ce1302dcd39153b9569e367fe396e352b0a34350039c WatchSource:0}: Error finding container d03007412f277fbe2703ce1302dcd39153b9569e367fe396e352b0a34350039c: Status 404 returned error can't find the container with id d03007412f277fbe2703ce1302dcd39153b9569e367fe396e352b0a34350039c
Sep 11 14:07:49.202485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:49.202466    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 14:07:49.891008 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:49.890968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d" event={"ID":"b454f9b7-f90c-459a-b74b-b8a56d454448","Type":"ContainerStarted","Data":"fde9b278bda2075c5a527dac48f394d2f958d81396b8924ff9b581cb40da0270"}
Sep 11 14:07:49.891466 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:49.891084    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:07:49.892508 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:49.892478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84" event={"ID":"f6a0a2b5-adde-4408-8f6f-aded4c762bf8","Type":"ContainerStarted","Data":"d03007412f277fbe2703ce1302dcd39153b9569e367fe396e352b0a34350039c"}
Sep 11 14:07:49.902022 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:49.901952    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:49.910948 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:49.910903    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d" podStartSLOduration=2.282125278 podStartE2EDuration="4.91088455s" totalImagesPullingTime="2.628759272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:49.908302964 +0000 UTC m=+603.871917940" watchObservedRunningTime="2026-09-11 14:07:49.91088455 +0000 UTC m=+603.874499525"
Sep 11 14:07:50.002699 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:50.002663    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:50.527010 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:50.526964    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:50.630276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:50.630239    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:50.928766 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:50.928688    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:51.010649 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:51.010619    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:53.913235 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:53.913159    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84" event={"ID":"f6a0a2b5-adde-4408-8f6f-aded4c762bf8","Type":"ContainerStarted","Data":"6d1094ff18a6e1f8cf19d34598320a25ccd41897abcbe268f87a50940c290dba"}
Sep 11 14:07:53.934971 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:53.934930    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-frt84" podStartSLOduration=2.431350002 podStartE2EDuration="6.934914647s" totalImagesPullingTime="4.503564645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:53.934898786 +0000 UTC m=+607.898513757" watchObservedRunningTime="2026-09-11 14:07:53.934914647 +0000 UTC m=+607.898529621"
Sep 11 14:07:59.177048 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:59.176988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:07:59.273808 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:07:59.273766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:00.899639 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:00.899604    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7vf8d"
Sep 11 14:08:02.158237 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:02.158202    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:02.255374 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:02.255338    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:04.637435 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:04.637399    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:04.740219 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:04.740189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:11.705690 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:11.705597    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:11.901043 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:11.900999    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:12.078461 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:12.078389    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:20.437152 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:20.437111    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:20.536404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:20.536376    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:23.445287 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:23.445252    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:23.547095 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:23.547068    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:25.896304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:25.896272    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:27.285250 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:27.285213    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:29.674205 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:29.674169    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:29.778494 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:29.778455    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:31.920404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:31.920369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:31.997919 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:31.997872    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:32.251329 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.251293    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:32.357759 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.357730    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:32.460340 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.460303    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:32.559163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.559085    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:32.705479 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.705449    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:08:32.724649 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.724623    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:08:32.724787 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.724722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.727333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.727305    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 14:08:32.741539 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.741515    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:08:32.810207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.810123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b05281a-cab7-40da-9312-3d9805aaec42-config-file\") pod \"limitador-limitador-84994f7ffc-pft78\" (UID: \"0b05281a-cab7-40da-9312-3d9805aaec42\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.810367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.810244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msg72\" (UniqueName: \"kubernetes.io/projected/0b05281a-cab7-40da-9312-3d9805aaec42-kube-api-access-msg72\") pod \"limitador-limitador-84994f7ffc-pft78\" (UID: \"0b05281a-cab7-40da-9312-3d9805aaec42\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.840395 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.840361    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:32.911160 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.911133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b05281a-cab7-40da-9312-3d9805aaec42-config-file\") pod \"limitador-limitador-84994f7ffc-pft78\" (UID: \"0b05281a-cab7-40da-9312-3d9805aaec42\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.911274 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.911211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msg72\" (UniqueName: \"kubernetes.io/projected/0b05281a-cab7-40da-9312-3d9805aaec42-kube-api-access-msg72\") pod \"limitador-limitador-84994f7ffc-pft78\" (UID: \"0b05281a-cab7-40da-9312-3d9805aaec42\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.911703 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.911686    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b05281a-cab7-40da-9312-3d9805aaec42-config-file\") pod \"limitador-limitador-84994f7ffc-pft78\" (UID: \"0b05281a-cab7-40da-9312-3d9805aaec42\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.922408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.922386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msg72\" (UniqueName: \"kubernetes.io/projected/0b05281a-cab7-40da-9312-3d9805aaec42-kube-api-access-msg72\") pod \"limitador-limitador-84994f7ffc-pft78\" (UID: \"0b05281a-cab7-40da-9312-3d9805aaec42\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:32.940898 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:32.940874    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:33.035201 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.035181    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:33.110710 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.110677    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:33.206846 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.206814    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:33.207206 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.207188    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwm4z"]
Sep 11 14:08:33.209739 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.209724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:08:33.212790 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.212769    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-skzf8\""
Sep 11 14:08:33.213086 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.213067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwljl\" (UniqueName: \"kubernetes.io/projected/f2e136f1-5b31-4553-98f4-07b50843d639-kube-api-access-lwljl\") pod \"authorino-59c974bd48-qwm4z\" (UID: \"f2e136f1-5b31-4553-98f4-07b50843d639\") " pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:08:33.261470 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.261445    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwm4z"]
Sep 11 14:08:33.302121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.302090    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:33.310197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.310149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:08:33.312567 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:08:33.312545    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b05281a_cab7_40da_9312_3d9805aaec42.slice/crio-ff1fe91e36aef454c91a3e13dfb00ad5b0db56f17f5db354369acb1d7db4a61b WatchSource:0}: Error finding container ff1fe91e36aef454c91a3e13dfb00ad5b0db56f17f5db354369acb1d7db4a61b: Status 404 returned error can't find the container with id ff1fe91e36aef454c91a3e13dfb00ad5b0db56f17f5db354369acb1d7db4a61b
Sep 11 14:08:33.313991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.313972    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwljl\" (UniqueName: \"kubernetes.io/projected/f2e136f1-5b31-4553-98f4-07b50843d639-kube-api-access-lwljl\") pod \"authorino-59c974bd48-qwm4z\" (UID: \"f2e136f1-5b31-4553-98f4-07b50843d639\") " pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:08:33.322760 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.322737    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwljl\" (UniqueName: \"kubernetes.io/projected/f2e136f1-5b31-4553-98f4-07b50843d639-kube-api-access-lwljl\") pod \"authorino-59c974bd48-qwm4z\" (UID: \"f2e136f1-5b31-4553-98f4-07b50843d639\") " pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:08:33.453668 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.453642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:33.519766 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.519745    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:08:33.562283 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.562255    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:33.799165 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:33.799143    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwm4z"]
Sep 11 14:08:33.801186 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:08:33.801143    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2e136f1_5b31_4553_98f4_07b50843d639.slice/crio-f64c5ff6ef7564ba17d0a784f0fc4aad6857fa97e27e432831e5e8c75a627697 WatchSource:0}: Error finding container f64c5ff6ef7564ba17d0a784f0fc4aad6857fa97e27e432831e5e8c75a627697: Status 404 returned error can't find the container with id f64c5ff6ef7564ba17d0a784f0fc4aad6857fa97e27e432831e5e8c75a627697
Sep 11 14:08:34.079590 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:34.079563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78" event={"ID":"0b05281a-cab7-40da-9312-3d9805aaec42","Type":"ContainerStarted","Data":"ff1fe91e36aef454c91a3e13dfb00ad5b0db56f17f5db354369acb1d7db4a61b"}
Sep 11 14:08:34.080623 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:34.080592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwm4z" event={"ID":"f2e136f1-5b31-4553-98f4-07b50843d639","Type":"ContainerStarted","Data":"f64c5ff6ef7564ba17d0a784f0fc4aad6857fa97e27e432831e5e8c75a627697"}
Sep 11 14:08:35.088566 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:35.088471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78" event={"ID":"0b05281a-cab7-40da-9312-3d9805aaec42","Type":"ContainerStarted","Data":"8738bd8d86c92a52eae7c4568ba55536892b87a3a068110595598ef92c41e8af"}
Sep 11 14:08:35.089076 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:35.088738    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:36.094700 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:36.094650    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwm4z" event={"ID":"f2e136f1-5b31-4553-98f4-07b50843d639","Type":"ContainerStarted","Data":"d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516"}
Sep 11 14:08:36.113690 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:36.113643    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qwm4z" podStartSLOduration=1.752915657 podStartE2EDuration="3.113627835s" totalImagesPullingTime="1.360712178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:36.112115602 +0000 UTC m=+650.075730575" watchObservedRunningTime="2026-09-11 14:08:36.113627835 +0000 UTC m=+650.077242809"
Sep 11 14:08:36.114092 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:36.114065    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78" podStartSLOduration=3.378900926 podStartE2EDuration="4.114057833s" totalImagesPullingTime="735.156907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:35.110132453 +0000 UTC m=+649.073747427" watchObservedRunningTime="2026-09-11 14:08:36.114057833 +0000 UTC m=+650.077672808"
Sep 11 14:08:41.904488 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:41.904449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:42.026230 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:42.026197    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:45.021556 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:45.021511    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:45.410925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:45.410835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:46.095702 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:46.095672    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-pft78"
Sep 11 14:08:47.764533 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:47.764501    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:47.979441 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:47.979401    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:48.489334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:48.489305    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:48.595227 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:48.595195    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:48.692228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:48.692189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:08:48.985182 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:08:48.985148    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:01.656971 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:01.656929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:01.883868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:01.883835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:04.425955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:04.425921    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwm4z"]
Sep 11 14:09:04.426334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:04.426095    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qwm4z" podUID="f2e136f1-5b31-4553-98f4-07b50843d639" containerName="authorino" containerID="cri-o://d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516" gracePeriod=30
Sep 11 14:09:04.663482 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:04.663456    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:09:04.738604 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:04.738581    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwljl\" (UniqueName: \"kubernetes.io/projected/f2e136f1-5b31-4553-98f4-07b50843d639-kube-api-access-lwljl\") pod \"f2e136f1-5b31-4553-98f4-07b50843d639\" (UID: \"f2e136f1-5b31-4553-98f4-07b50843d639\") "
Sep 11 14:09:04.740592 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:04.740570    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e136f1-5b31-4553-98f4-07b50843d639-kube-api-access-lwljl" (OuterVolumeSpecName: "kube-api-access-lwljl") pod "f2e136f1-5b31-4553-98f4-07b50843d639" (UID: "f2e136f1-5b31-4553-98f4-07b50843d639"). InnerVolumeSpecName "kube-api-access-lwljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:09:04.839316 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:04.839291    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwljl\" (UniqueName: \"kubernetes.io/projected/f2e136f1-5b31-4553-98f4-07b50843d639-kube-api-access-lwljl\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:09:05.231200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.231169    2558 generic.go:356] "Generic (PLEG): container finished" podID="f2e136f1-5b31-4553-98f4-07b50843d639" containerID="d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516" exitCode=0
Sep 11 14:09:05.231332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.231220    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwm4z"
Sep 11 14:09:05.231332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.231254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwm4z" event={"ID":"f2e136f1-5b31-4553-98f4-07b50843d639","Type":"ContainerDied","Data":"d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516"}
Sep 11 14:09:05.231332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.231296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwm4z" event={"ID":"f2e136f1-5b31-4553-98f4-07b50843d639","Type":"ContainerDied","Data":"f64c5ff6ef7564ba17d0a784f0fc4aad6857fa97e27e432831e5e8c75a627697"}
Sep 11 14:09:05.231332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.231314    2558 scope.go:117] "RemoveContainer" containerID="d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516"
Sep 11 14:09:05.239599 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.239578    2558 scope.go:117] "RemoveContainer" containerID="d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516"
Sep 11 14:09:05.239882 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:09:05.239863    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516\": container with ID starting with d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516 not found: ID does not exist" containerID="d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516"
Sep 11 14:09:05.239929 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.239890    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516"} err="failed to get container status \"d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516\": rpc error: code = NotFound desc = could not find container \"d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516\": container with ID starting with d37754de89e13c9d6131814e0fbc556d1e26df4c3607a40ebab89766a1efb516 not found: ID does not exist"
Sep 11 14:09:05.255719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.255698    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwm4z"]
Sep 11 14:09:05.261982 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:05.261961    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwm4z"]
Sep 11 14:09:06.502965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:06.502933    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e136f1-5b31-4553-98f4-07b50843d639" path="/var/lib/kubelet/pods/f2e136f1-5b31-4553-98f4-07b50843d639/volumes"
Sep 11 14:09:09.671564 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:09.671530    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:09.759542 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:09.759490    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:09.860833 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:09.860806    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:09.938747 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:09.938680    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:10.015386 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:10.015359    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:13.227043 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:13.226998    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:13.409185 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:13.409152    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:34.950067 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:34.950007    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:43.974508 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:43.974428    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:44.073156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:44.073121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:45.578743 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:45.578710    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:45.786997 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:45.786962    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:46.024591 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:46.024558    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:46.582931 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:46.582890    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:46.816478 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:46.816395    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:46.942016 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:46.941983    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:47.158058 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:47.157968    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:09:56.184482 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:56.184450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:09:56.268946 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:09:56.268916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:02.677501 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:02.677469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:03.084228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:03.084194    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:10:21.688767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:21.688729    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:22.360167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:22.360135    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:10:30.675523 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.675487    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"]
Sep 11 14:10:30.676732 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.676700    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e136f1-5b31-4553-98f4-07b50843d639" containerName="authorino"
Sep 11 14:10:30.676732 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.676730    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e136f1-5b31-4553-98f4-07b50843d639" containerName="authorino"
Sep 11 14:10:30.676914 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.676898    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e136f1-5b31-4553-98f4-07b50843d639" containerName="authorino"
Sep 11 14:10:30.681064 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.681021    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.684576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.684553    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 14:10:30.685456 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.685437    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 14:10:30.685561 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.685475    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7lnrv\""
Sep 11 14:10:30.685561 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.685503    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 14:10:30.711791 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.711764    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"]
Sep 11 14:10:30.745878 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.745852    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:30.766870 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.766850    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.766947 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.766885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.766947 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.766904    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9lw\" (UniqueName: \"kubernetes.io/projected/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-kube-api-access-5g9lw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.767052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.766955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.767052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.766970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.767052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.766987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868086 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9lw\" (UniqueName: \"kubernetes.io/projected/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-kube-api-access-5g9lw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868502 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868563 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.868601 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.868568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.870422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.870407    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.870688 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.870670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.877214 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.877191    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9lw\" (UniqueName: \"kubernetes.io/projected/19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2-kube-api-access-5g9lw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96\" (UID: \"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:30.991093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:30.991071    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:31.267366 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:31.267342    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"]
Sep 11 14:10:31.269436 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:10:31.269410    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f8cf4a_b0d8_41a6_aa2f_aee8ef2bd1d2.slice/crio-d5a1883f3c9cc41bd5e89f7fb67ba61a7c271c8b04d9a4095c7e83b28cdf3c94 WatchSource:0}: Error finding container d5a1883f3c9cc41bd5e89f7fb67ba61a7c271c8b04d9a4095c7e83b28cdf3c94: Status 404 returned error can't find the container with id d5a1883f3c9cc41bd5e89f7fb67ba61a7c271c8b04d9a4095c7e83b28cdf3c94
Sep 11 14:10:31.606638 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:31.606554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96" event={"ID":"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2","Type":"ContainerStarted","Data":"d5a1883f3c9cc41bd5e89f7fb67ba61a7c271c8b04d9a4095c7e83b28cdf3c94"}
Sep 11 14:10:31.609872 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:31.609655    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:10:32.612167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:32.612084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96" event={"ID":"19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2","Type":"ContainerStarted","Data":"72d7d3e92f47f33d8019e09739ed001441cb9c99502d0b29f1ae336acf236120"}
Sep 11 14:10:32.634490 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:32.634444    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96" podStartSLOduration=1.593788073 podStartE2EDuration="2.634429026s" totalImagesPullingTime="1.040640953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:32.632959914 +0000 UTC m=+766.596574887" watchObservedRunningTime="2026-09-11 14:10:32.634429026 +0000 UTC m=+766.598044000"
Sep 11 14:10:38.022494 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:38.022458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:39.263103 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:39.263072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:10:40.992053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:40.992003    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:40.992477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:40.992072    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:41.004644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:41.004618    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:41.663496 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:41.663470    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96"
Sep 11 14:10:44.919474 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:44.919434    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:52.470084 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:52.470044    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:10:54.093836 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:10:54.093798    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:11:14.681102 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.681015    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"]
Sep 11 14:11:14.683473 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.683458    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.686594 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.686574    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 14:11:14.698213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.698195    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"]
Sep 11 14:11:14.785366 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.785342    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:11:14.802192 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.802162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.802322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.802207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnj5\" (UniqueName: \"kubernetes.io/projected/0eaad85b-7061-4197-a163-7640ab1cdcca-kube-api-access-rnnj5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.802322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.802274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaad85b-7061-4197-a163-7640ab1cdcca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.802322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.802301    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.802453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.802324    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.802453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.802344    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903495 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaad85b-7061-4197-a163-7640ab1cdcca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903529    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903762 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903762 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnj5\" (UniqueName: \"kubernetes.io/projected/0eaad85b-7061-4197-a163-7640ab1cdcca-kube-api-access-rnnj5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.903973 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.904081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.903952    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.904081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.904016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.905645 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.905617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaad85b-7061-4197-a163-7640ab1cdcca-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.905909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.905887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaad85b-7061-4197-a163-7640ab1cdcca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.912464 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.912424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnj5\" (UniqueName: \"kubernetes.io/projected/0eaad85b-7061-4197-a163-7640ab1cdcca-kube-api-access-rnnj5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bx5r7\" (UID: \"0eaad85b-7061-4197-a163-7640ab1cdcca\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:14.992964 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:14.992935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:15.284739 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:15.284681    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"]
Sep 11 14:11:15.288690 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:11:15.288658    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaad85b_7061_4197_a163_7640ab1cdcca.slice/crio-fd33d2e1c33cfeb44400a0b93bb8e11df389b09ce024239a8eb10de9c1533e10 WatchSource:0}: Error finding container fd33d2e1c33cfeb44400a0b93bb8e11df389b09ce024239a8eb10de9c1533e10: Status 404 returned error can't find the container with id fd33d2e1c33cfeb44400a0b93bb8e11df389b09ce024239a8eb10de9c1533e10
Sep 11 14:11:15.290233 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:15.290213    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:11:15.797702 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:15.797672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7" event={"ID":"0eaad85b-7061-4197-a163-7640ab1cdcca","Type":"ContainerStarted","Data":"9f52d34ccd84fdf7598cb859b4f545c8ae1593ab470ffb2277d9cae39fca7385"}
Sep 11 14:11:15.798058 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:15.797721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7" event={"ID":"0eaad85b-7061-4197-a163-7640ab1cdcca","Type":"ContainerStarted","Data":"fd33d2e1c33cfeb44400a0b93bb8e11df389b09ce024239a8eb10de9c1533e10"}
Sep 11 14:11:15.820420 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:15.820379    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7" podStartSLOduration=1.654343929 podStartE2EDuration="1.820363712s" totalImagesPullingTime="166.019783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:15.81923289 +0000 UTC m=+809.782847863" watchObservedRunningTime="2026-09-11 14:11:15.820363712 +0000 UTC m=+809.783978681"
Sep 11 14:11:16.572198 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:16.572166    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:11:24.004218 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:24.004184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:11:24.108522 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:24.108490    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:11:24.993758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:24.993726    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:24.994016 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:24.993786    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:25.006373 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:25.006349    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:25.854565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:25.854531    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bx5r7"
Sep 11 14:11:28.459568 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.459535    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"]
Sep 11 14:11:28.462795 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.462778    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.467876 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.467853    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 14:11:28.475239 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.475213    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"]
Sep 11 14:11:28.499863 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.499832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.499998 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.499918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.499998 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.499975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.500161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.500022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6dg\" (UniqueName: \"kubernetes.io/projected/a491f53d-a797-497d-9cf6-ed268b139262-kube-api-access-fj6dg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.500161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.500061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.500161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.500138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a491f53d-a797-497d-9cf6-ed268b139262-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.500473 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.500449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:11:28.601114 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601090    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601176    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6dg\" (UniqueName: \"kubernetes.io/projected/a491f53d-a797-497d-9cf6-ed268b139262-kube-api-access-fj6dg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a491f53d-a797-497d-9cf6-ed268b139262-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601594 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.601594 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.601559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.603250 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.603231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a491f53d-a797-497d-9cf6-ed268b139262-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.603335 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.603319    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a491f53d-a797-497d-9cf6-ed268b139262-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.610904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.610883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6dg\" (UniqueName: \"kubernetes.io/projected/a491f53d-a797-497d-9cf6-ed268b139262-kube-api-access-fj6dg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5\" (UID: \"a491f53d-a797-497d-9cf6-ed268b139262\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:28.772438 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:28.772379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:29.064761 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:29.064687    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"]
Sep 11 14:11:29.067463 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:11:29.067430    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda491f53d_a797_497d_9cf6_ed268b139262.slice/crio-d1cbeec1aa5bec9fa083b4cced5c01a602f9b821fe6ed600e008e487a357c9ff WatchSource:0}: Error finding container d1cbeec1aa5bec9fa083b4cced5c01a602f9b821fe6ed600e008e487a357c9ff: Status 404 returned error can't find the container with id d1cbeec1aa5bec9fa083b4cced5c01a602f9b821fe6ed600e008e487a357c9ff
Sep 11 14:11:29.863519 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:29.863485    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5" event={"ID":"a491f53d-a797-497d-9cf6-ed268b139262","Type":"ContainerStarted","Data":"3447321108cd2f56ee8415852a01398a7106207f88062eb2c61066c7c3a3e853"}
Sep 11 14:11:29.863902 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:29.863534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5" event={"ID":"a491f53d-a797-497d-9cf6-ed268b139262","Type":"ContainerStarted","Data":"d1cbeec1aa5bec9fa083b4cced5c01a602f9b821fe6ed600e008e487a357c9ff"}
Sep 11 14:11:29.888829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:29.888784    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5" podStartSLOduration=1.702944011 podStartE2EDuration="1.888767043s" totalImagesPullingTime="185.823032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:29.888155577 +0000 UTC m=+823.851770563" watchObservedRunningTime="2026-09-11 14:11:29.888767043 +0000 UTC m=+823.852382015"
Sep 11 14:11:30.979668 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:30.979632    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:11:38.773383 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:38.773343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:38.773833 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:38.773418    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:38.785758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:38.785729    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:38.913771 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:38.913747    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5"
Sep 11 14:11:46.233151 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:46.233105    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:11:46.339309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:11:46.339272    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:12:46.471511 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:12:46.471444    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:12:46.478270 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:12:46.478244    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:12:55.381369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:12:55.381335    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:12:55.480064 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:12:55.480018    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:12:56.652288 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:12:56.652254    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:12:56.743336 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:12:56.743301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:12.926946 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:12.926916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:13.068156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:13.068125    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:13.391079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:13.391049    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:13.499052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:13.499010    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:14.414068 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:14.414009    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:14.510920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:14.510897    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:14.876400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:14.876370    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:15.003406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.003381    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7717-ba7103"]
Sep 11 14:13:15.006607 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.006588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:15.010744 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.010728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4lz74\""
Sep 11 14:13:15.018634 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.018614    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7717-ba7103"]
Sep 11 14:13:15.105173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.105151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwtf\" (UniqueName: \"kubernetes.io/projected/5399fc6a-5091-4368-a8b1-2671b0d33744-kube-api-access-6vwtf\") pod \"test-curl-7717-ba7103\" (UID: \"5399fc6a-5091-4368-a8b1-2671b0d33744\") " pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:15.205828 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.205803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwtf\" (UniqueName: \"kubernetes.io/projected/5399fc6a-5091-4368-a8b1-2671b0d33744-kube-api-access-6vwtf\") pod \"test-curl-7717-ba7103\" (UID: \"5399fc6a-5091-4368-a8b1-2671b0d33744\") " pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:15.219289 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.219262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwtf\" (UniqueName: \"kubernetes.io/projected/5399fc6a-5091-4368-a8b1-2671b0d33744-kube-api-access-6vwtf\") pod \"test-curl-7717-ba7103\" (UID: \"5399fc6a-5091-4368-a8b1-2671b0d33744\") " pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:15.315638 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.315617    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:15.588989 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:15.588883    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7717-ba7103"]
Sep 11 14:13:15.592997 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:13:15.592966    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5399fc6a_5091_4368_a8b1_2671b0d33744.slice/crio-2c9d90fe77f9908057d6ef1412efd77f7bbe8d8e70a92b791c3ba0fd8c822cba WatchSource:0}: Error finding container 2c9d90fe77f9908057d6ef1412efd77f7bbe8d8e70a92b791c3ba0fd8c822cba: Status 404 returned error can't find the container with id 2c9d90fe77f9908057d6ef1412efd77f7bbe8d8e70a92b791c3ba0fd8c822cba
Sep 11 14:13:16.349141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:16.349110    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7717-ba7103" event={"ID":"5399fc6a-5091-4368-a8b1-2671b0d33744","Type":"ContainerStarted","Data":"2c9d90fe77f9908057d6ef1412efd77f7bbe8d8e70a92b791c3ba0fd8c822cba"}
Sep 11 14:13:16.923675 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:16.923643    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:17.068379 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:17.068342    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:13:17.101854 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:17.101826    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:18.362565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:18.362529    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7717-ba7103" event={"ID":"5399fc6a-5091-4368-a8b1-2671b0d33744","Type":"ContainerStarted","Data":"8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd"}
Sep 11 14:13:18.384993 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:18.384945    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7717-ba7103" podStartSLOduration=2.383888618 podStartE2EDuration="4.384931481s" totalImagesPullingTime="2.001042863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:18.384461215 +0000 UTC m=+932.348076186" watchObservedRunningTime="2026-09-11 14:13:18.384931481 +0000 UTC m=+932.348546486"
Sep 11 14:13:18.392593 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:18.392564    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:18.509089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:18.509060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:18.854164 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:18.854132    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:19.118986 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.118892    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:19.123218 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.123195    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7717-ba7103"]
Sep 11 14:13:19.144540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.144521    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7717-ba7103"]
Sep 11 14:13:19.237071 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.237021    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:19.276734 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.276710    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7717-d0c56c"]
Sep 11 14:13:19.277078 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.277064    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" containerName="test-curl-7717-ba7103"
Sep 11 14:13:19.277120 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.277079    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" containerName="test-curl-7717-ba7103"
Sep 11 14:13:19.277164 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.277147    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" containerName="test-curl-7717-ba7103"
Sep 11 14:13:19.280147 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.280133    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:19.292463 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.292442    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7717-d0c56c"]
Sep 11 14:13:19.367269 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:19.367246    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7717-ba7103" container="test-curl-7717-ba7103"
Sep 11 14:13:19.440907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.440887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9gd\" (UniqueName: \"kubernetes.io/projected/216e3a68-1074-455d-b76a-3a01d404eb01-kube-api-access-sl9gd\") pod \"test-curl-7717-d0c56c\" (UID: \"216e3a68-1074-455d-b76a-3a01d404eb01\") " pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:19.542240 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.542212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9gd\" (UniqueName: \"kubernetes.io/projected/216e3a68-1074-455d-b76a-3a01d404eb01-kube-api-access-sl9gd\") pod \"test-curl-7717-d0c56c\" (UID: \"216e3a68-1074-455d-b76a-3a01d404eb01\") " pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:19.559711 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.559682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9gd\" (UniqueName: \"kubernetes.io/projected/216e3a68-1074-455d-b76a-3a01d404eb01-kube-api-access-sl9gd\") pod \"test-curl-7717-d0c56c\" (UID: \"216e3a68-1074-455d-b76a-3a01d404eb01\") " pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:19.588550 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.588531    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:19.862342 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:19.862322    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7717-d0c56c"]
Sep 11 14:13:19.864280 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:13:19.864245    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod216e3a68_1074_455d_b76a_3a01d404eb01.slice/crio-7da0f4ccd7fd80c2792586d5bb26ad99b877e18594d6579f01850dbcb547c096 WatchSource:0}: Error finding container 7da0f4ccd7fd80c2792586d5bb26ad99b877e18594d6579f01850dbcb547c096: Status 404 returned error can't find the container with id 7da0f4ccd7fd80c2792586d5bb26ad99b877e18594d6579f01850dbcb547c096
Sep 11 14:13:20.372735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:20.372705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7717-d0c56c" event={"ID":"216e3a68-1074-455d-b76a-3a01d404eb01","Type":"ContainerStarted","Data":"87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190"}
Sep 11 14:13:20.373104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:20.372750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7717-d0c56c" event={"ID":"216e3a68-1074-455d-b76a-3a01d404eb01","Type":"ContainerStarted","Data":"7da0f4ccd7fd80c2792586d5bb26ad99b877e18594d6579f01850dbcb547c096"}
Sep 11 14:13:20.373104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:20.372833    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7717-ba7103" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" containerName="test-curl-7717-ba7103" containerID="cri-o://8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd" gracePeriod=2
Sep 11 14:13:20.403041 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:20.402993    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7717-d0c56c" podStartSLOduration=1.402978918 podStartE2EDuration="1.402978918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:20.401887904 +0000 UTC m=+934.365502878" watchObservedRunningTime="2026-09-11 14:13:20.402978918 +0000 UTC m=+934.366593891"
Sep 11 14:13:20.404410 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:20.404387    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7717-ba7103\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:21.055998 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:21.055968    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7717-d0c56c"]
Sep 11 14:13:21.063675 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:21.063651    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7717-d0c56c"]
Sep 11 14:13:21.377416 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:21.377391    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7717-d0c56c" container="test-curl-7717-d0c56c"
Sep 11 14:13:21.630529 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:21.630455    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:22.034827 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.034804    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:22.287210 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.287144    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:22.382534 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.382506    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7717-d0c56c" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" containerName="test-curl-7717-d0c56c" containerID="cri-o://87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190" gracePeriod=2
Sep 11 14:13:22.387979 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:22.387956    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7717-d0c56c\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:22.487492 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.487452    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:22.565214 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.565193    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:22.667442 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.667420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vwtf\" (UniqueName: \"kubernetes.io/projected/5399fc6a-5091-4368-a8b1-2671b0d33744-kube-api-access-6vwtf\") pod \"5399fc6a-5091-4368-a8b1-2671b0d33744\" (UID: \"5399fc6a-5091-4368-a8b1-2671b0d33744\") "
Sep 11 14:13:22.669628 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.669606    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5399fc6a-5091-4368-a8b1-2671b0d33744-kube-api-access-6vwtf" (OuterVolumeSpecName: "kube-api-access-6vwtf") pod "5399fc6a-5091-4368-a8b1-2671b0d33744" (UID: "5399fc6a-5091-4368-a8b1-2671b0d33744"). InnerVolumeSpecName "kube-api-access-6vwtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:13:22.709536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.709515    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:22.768084 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:22.768059    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwtf\" (UniqueName: \"kubernetes.io/projected/5399fc6a-5091-4368-a8b1-2671b0d33744-kube-api-access-6vwtf\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:13:23.172856 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:23.172829    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:23.388080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:23.388048    2558 generic.go:356] "Generic (PLEG): container finished" podID="5399fc6a-5091-4368-a8b1-2671b0d33744" containerID="8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd" exitCode=137
Sep 11 14:13:23.388428 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:23.388098    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:23.388428 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:23.388133    2558 scope.go:117] "RemoveContainer" containerID="8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd"
Sep 11 14:13:23.397067 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:23.397050    2558 scope.go:117] "RemoveContainer" containerID="8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd"
Sep 11 14:13:23.397314 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:23.397294    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd\": container with ID starting with 8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd not found: ID does not exist" containerID="8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd"
Sep 11 14:13:23.397363 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:23.397323    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd"} err="failed to get container status \"8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd\": rpc error: code = NotFound desc = could not find container \"8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd\": container with ID starting with 8be5521d0d3e79895019d9536d4a933176899a134f456094b0c12bfdeece53cd not found: ID does not exist"
Sep 11 14:13:23.398365 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:23.398342    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7717-ba7103\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" pod="opendatahub/test-curl-7717-ba7103"
Sep 11 14:13:24.352353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.352320    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:24.503358 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.503327    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5399fc6a-5091-4368-a8b1-2671b0d33744" path="/var/lib/kubelet/pods/5399fc6a-5091-4368-a8b1-2671b0d33744/volumes"
Sep 11 14:13:24.542837 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.542817    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:24.685894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.685844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl9gd\" (UniqueName: \"kubernetes.io/projected/216e3a68-1074-455d-b76a-3a01d404eb01-kube-api-access-sl9gd\") pod \"216e3a68-1074-455d-b76a-3a01d404eb01\" (UID: \"216e3a68-1074-455d-b76a-3a01d404eb01\") "
Sep 11 14:13:24.687789 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.687760    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216e3a68-1074-455d-b76a-3a01d404eb01-kube-api-access-sl9gd" (OuterVolumeSpecName: "kube-api-access-sl9gd") pod "216e3a68-1074-455d-b76a-3a01d404eb01" (UID: "216e3a68-1074-455d-b76a-3a01d404eb01"). InnerVolumeSpecName "kube-api-access-sl9gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:13:24.786782 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.786757    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9gd\" (UniqueName: \"kubernetes.io/projected/216e3a68-1074-455d-b76a-3a01d404eb01-kube-api-access-sl9gd\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:13:24.879245 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.879217    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:24.978929 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:24.978898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:25.399152 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.399060    2558 generic.go:356] "Generic (PLEG): container finished" podID="216e3a68-1074-455d-b76a-3a01d404eb01" containerID="87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190" exitCode=137
Sep 11 14:13:25.399152 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.399102    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:25.399376 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.399157    2558 scope.go:117] "RemoveContainer" containerID="87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190"
Sep 11 14:13:25.408434 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.408409    2558 scope.go:117] "RemoveContainer" containerID="87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190"
Sep 11 14:13:25.408721 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:25.408700    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190\": container with ID starting with 87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190 not found: ID does not exist" containerID="87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190"
Sep 11 14:13:25.408780 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.408731    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190"} err="failed to get container status \"87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190\": rpc error: code = NotFound desc = could not find container \"87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190\": container with ID starting with 87662f130ec3a7fb86643a466dfa28f24aa0ecfe4063b4e3a7511c0b9941a190 not found: ID does not exist"
Sep 11 14:13:25.409793 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:25.409765    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7717-d0c56c\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:25.483536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.483502    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:25.645279 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.645249    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:25.798196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.798166    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:25.922065 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:25.922041    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:26.502395 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:26.502364    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7717-d0c56c\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" pod="opendatahub/test-curl-7717-d0c56c"
Sep 11 14:13:26.503360 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:26.503340    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" path="/var/lib/kubelet/pods/216e3a68-1074-455d-b76a-3a01d404eb01/volumes"
Sep 11 14:13:27.333782 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:27.333753    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:28.095333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.095303    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"]
Sep 11 14:13:28.095650 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.095638    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" containerName="test-curl-7717-d0c56c"
Sep 11 14:13:28.095694 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.095652    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" containerName="test-curl-7717-d0c56c"
Sep 11 14:13:28.095732 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.095699    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="216e3a68-1074-455d-b76a-3a01d404eb01" containerName="test-curl-7717-d0c56c"
Sep 11 14:13:28.098738 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.098723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.115717 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.115700    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1dc719b7\""
Sep 11 14:13:28.116783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.116771    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1dc719b7-dockercfg-8lvg5\""
Sep 11 14:13:28.121850 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.121812    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"]
Sep 11 14:13:28.212595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.212565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6hhq\" (UniqueName: \"kubernetes.io/projected/761e0ee4-031e-45a9-a834-da73e4d62c5c-kube-api-access-x6hhq\") pod \"payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.212727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.212631    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761e0ee4-031e-45a9-a834-da73e4d62c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.313950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.313922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761e0ee4-031e-45a9-a834-da73e4d62c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.314097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.314004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6hhq\" (UniqueName: \"kubernetes.io/projected/761e0ee4-031e-45a9-a834-da73e4d62c5c-kube-api-access-x6hhq\") pod \"payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.314600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.314580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761e0ee4-031e-45a9-a834-da73e4d62c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.325109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.325088    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6hhq\" (UniqueName: \"kubernetes.io/projected/761e0ee4-031e-45a9-a834-da73e4d62c5c-kube-api-access-x6hhq\") pod \"payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.408642 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.408586    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:28.529766 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.529734    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"]
Sep 11 14:13:28.541838 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.541715    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.544617 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.544590    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"]
Sep 11 14:13:28.544807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.544731    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:13:28.544919 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.544899    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:13:28.544979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.544922    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:13:28.545199 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.545173    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-2a21d9-serving-cert\""
Sep 11 14:13:28.545302 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.545222    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktm9c\""
Sep 11 14:13:28.682505 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.682483    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"]
Sep 11 14:13:28.684268 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:13:28.684241    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761e0ee4_031e_45a9_a834_da73e4d62c5c.slice/crio-8a38dca6647d82b5034f237be4115fabd2787e8ed6f0e75dd8e0b55f8fa0f84e WatchSource:0}: Error finding container 8a38dca6647d82b5034f237be4115fabd2787e8ed6f0e75dd8e0b55f8fa0f84e: Status 404 returned error can't find the container with id 8a38dca6647d82b5034f237be4115fabd2787e8ed6f0e75dd8e0b55f8fa0f84e
Sep 11 14:13:28.718979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.718958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrcq\" (UniqueName: \"kubernetes.io/projected/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-kube-api-access-hdrcq\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.719097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.718986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-openshift-service-ca\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.719097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.719005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-metrics-certs\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.719097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.719077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-maas-api-tls\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.819957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.819926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrcq\" (UniqueName: \"kubernetes.io/projected/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-kube-api-access-hdrcq\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.820081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.819965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-openshift-service-ca\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.820081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.819989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-metrics-certs\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.820081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.820070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-maas-api-tls\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.820565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.820544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-openshift-service-ca\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.822276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.822258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-maas-api-tls\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.822370 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.822338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-metrics-certs\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.832357 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.832339    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrcq\" (UniqueName: \"kubernetes.io/projected/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-kube-api-access-hdrcq\") pod \"maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.856990 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.856973    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:28.883363 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:28.883326    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:29.128501 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:29.128454    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"]
Sep 11 14:13:29.130496 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:13:29.130475    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4b52fc_1361_4d55_b5c8_a05ce9fbbfb0.slice/crio-3b3ade7556cb800c10e27e70eac7240d895a3bff25ce7a6643acf60adb763828 WatchSource:0}: Error finding container 3b3ade7556cb800c10e27e70eac7240d895a3bff25ce7a6643acf60adb763828: Status 404 returned error can't find the container with id 3b3ade7556cb800c10e27e70eac7240d895a3bff25ce7a6643acf60adb763828
Sep 11 14:13:29.237802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:29.237778    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:29.352475 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:29.352448    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:29.418948 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:29.418878    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" event={"ID":"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0","Type":"ContainerStarted","Data":"3b3ade7556cb800c10e27e70eac7240d895a3bff25ce7a6643acf60adb763828"}
Sep 11 14:13:29.420605 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:29.420577    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9" event={"ID":"761e0ee4-031e-45a9-a834-da73e4d62c5c","Type":"ContainerStarted","Data":"8a38dca6647d82b5034f237be4115fabd2787e8ed6f0e75dd8e0b55f8fa0f84e"}
Sep 11 14:13:29.990584 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:29.990550    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:30.990061 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:30.990000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:31.594903 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:31.594874    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:31.827966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:31.827933    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:31.944900 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:31.944868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:32.042754 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:32.042717    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:32.157517 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:32.157483    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:32.282295 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:32.282201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:32.408356 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:32.408324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:32.514546 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:32.514520    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:32.791787 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:32.791760    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:13:33.101089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.100991    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:33.442744 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.442707    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" event={"ID":"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0","Type":"ContainerStarted","Data":"43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4"}
Sep 11 14:13:33.442919 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.442859    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:33.444191 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.444170    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9" event={"ID":"761e0ee4-031e-45a9-a834-da73e4d62c5c","Type":"ContainerStarted","Data":"1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172"}
Sep 11 14:13:33.444307 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.444286    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:33.475189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.475144    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" podStartSLOduration=2.136076364 podStartE2EDuration="5.475126117s" totalImagesPullingTime="3.339049753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:33.472476758 +0000 UTC m=+947.436091730" watchObservedRunningTime="2026-09-11 14:13:33.475126117 +0000 UTC m=+947.438741102"
Sep 11 14:13:33.505757 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:33.505721    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9" podStartSLOduration=1.764876747 podStartE2EDuration="5.505708051s" totalImagesPullingTime="3.740831304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:33.501402774 +0000 UTC m=+947.465017746" watchObservedRunningTime="2026-09-11 14:13:33.505708051 +0000 UTC m=+947.469323024"
Sep 11 14:13:35.052455 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:35.052425    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:35.603629 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:35.603593    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:36.306550 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:36.306514    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:36.422197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:36.422166    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:36.957070 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:36.957042    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:37.061813 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:37.061792    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:37.976770 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:37.976741    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:38.989499 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:38.989469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:39.135707 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:39.135680    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:39.254313 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:39.254247    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:39.455372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:39.455349    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:13:40.107115 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.107081    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"]
Sep 11 14:13:40.112060 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.112020    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.114706 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.114687    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c6156e-serving-cert\""
Sep 11 14:13:40.128142 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.128123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"]
Sep 11 14:13:40.173796 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.173771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:40.208307 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.208288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.208394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.208345    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-metrics-certs\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.208394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.208364    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.208394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.208390    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xj5\" (UniqueName: \"kubernetes.io/projected/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-kube-api-access-p5xj5\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.308311 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.308280    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:40.309253 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.309224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xj5\" (UniqueName: \"kubernetes.io/projected/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-kube-api-access-p5xj5\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.309387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.309262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.309387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.309323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-metrics-certs\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.309387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.309341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.309616 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:40.309432    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-serving-cert: secret "maas-api-e2e-inf-b-c6156e-serving-cert" not found
Sep 11 14:13:40.309616 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:40.309483    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls podName:6dfd9a8e-ecd0-467c-a382-5c2ee44124b6 nodeName:}" failed. No retries permitted until 2026-09-11 14:13:40.809468355 +0000 UTC m=+954.773083306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls") pod "maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" (UID: "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6") : secret "maas-api-e2e-inf-b-c6156e-serving-cert" not found
Sep 11 14:13:40.310004 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.309985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.311714 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.311695    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-metrics-certs\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.321175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.321156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xj5\" (UniqueName: \"kubernetes.io/projected/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-kube-api-access-p5xj5\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.601653 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.601620    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:40.812150 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.812121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.814374 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.814352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:40.937327 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:40.937251    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:41.023450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.023417    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:41.037018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.036986    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:41.075106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.075041    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"]
Sep 11 14:13:41.079542 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.079523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.083564 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.083539    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c6156e\""
Sep 11 14:13:41.095640 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.095618    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"]
Sep 11 14:13:41.215309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.215287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.215587 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.215341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhbc\" (UniqueName: \"kubernetes.io/projected/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-kube-api-access-cfhbc\") pod \"payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.296894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.296872    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"]
Sep 11 14:13:41.299059 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:13:41.299012    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfd9a8e_ecd0_467c_a382_5c2ee44124b6.slice/crio-0c3c235e1a7a8587b44a181661825c2131bd99c35a4959740eb5c95d3e01b12d WatchSource:0}: Error finding container 0c3c235e1a7a8587b44a181661825c2131bd99c35a4959740eb5c95d3e01b12d: Status 404 returned error can't find the container with id 0c3c235e1a7a8587b44a181661825c2131bd99c35a4959740eb5c95d3e01b12d
Sep 11 14:13:41.315935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.315912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.316091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.315975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhbc\" (UniqueName: \"kubernetes.io/projected/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-kube-api-access-cfhbc\") pod \"payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.316611 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.316592    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.326297 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.326277    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhbc\" (UniqueName: \"kubernetes.io/projected/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-kube-api-access-cfhbc\") pod \"payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.393335 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.393308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:41.481411 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.481336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" event={"ID":"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6","Type":"ContainerStarted","Data":"0c3c235e1a7a8587b44a181661825c2131bd99c35a4959740eb5c95d3e01b12d"}
Sep 11 14:13:41.493694 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.493656    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:41.690867 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:41.690843    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"]
Sep 11 14:13:41.690994 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:13:41.690851    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea6c311_fc5d_4ae5_ae2d_cde5a46715f5.slice/crio-caf21c4d2b85cc541485a37967908a2ad6b025f728040dca874e864be1882391 WatchSource:0}: Error finding container caf21c4d2b85cc541485a37967908a2ad6b025f728040dca874e864be1882391: Status 404 returned error can't find the container with id caf21c4d2b85cc541485a37967908a2ad6b025f728040dca874e864be1882391
Sep 11 14:13:42.487674 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.487633    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" event={"ID":"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6","Type":"ContainerStarted","Data":"22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45"}
Sep 11 14:13:42.488115 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.487700    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:42.489239 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.489213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk" event={"ID":"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5","Type":"ContainerStarted","Data":"d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520"}
Sep 11 14:13:42.489343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.489253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk" event={"ID":"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5","Type":"ContainerStarted","Data":"caf21c4d2b85cc541485a37967908a2ad6b025f728040dca874e864be1882391"}
Sep 11 14:13:42.489343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.489282    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:42.508398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.508355    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" podStartSLOduration=1.949089356 podStartE2EDuration="2.508343775s" totalImagesPullingTime="559.254419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:42.507349596 +0000 UTC m=+956.470964569" watchObservedRunningTime="2026-09-11 14:13:42.508343775 +0000 UTC m=+956.471958746"
Sep 11 14:13:42.527906 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.527861    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk" podStartSLOduration=1.527848976 podStartE2EDuration="1.527848976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:42.52642768 +0000 UTC m=+956.490042657" watchObservedRunningTime="2026-09-11 14:13:42.527848976 +0000 UTC m=+956.491463949"
Sep 11 14:13:42.709822 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:42.709798    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:43.510516 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:43.510478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:44.367416 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:44.367378    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:44.451893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:44.451869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:45.393191 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:45.393160    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:46.496671 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:46.496640    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:46.614415 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:46.614372    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:46.718322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:46.718289    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:47.033877 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:47.033833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:48.188508 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:48.188475    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:48.503395 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:48.503368    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:13:48.610802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:48.610774    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:13:49.128590 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:49.128558    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:49.908426 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:49.908377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:50.093160 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:50.093123    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:51.042963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:51.042929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:52.079199 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:52.079171    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:53.066815 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:53.066785    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:53.494567 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:53.494536    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:13:53.616437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:53.616407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:53.725195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:53.725168    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:54.067591 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:54.067555    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:55.079336 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:55.079303    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:55.938741 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:55.938701    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:56.193533 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:56.193505    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:56.939215 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:56.939187    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:57.738191 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:57.738145    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:57.804992 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:57.804960    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"]
Sep 11 14:13:57.805290 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:57.805257    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9" podUID="761e0ee4-031e-45a9-a834-da73e4d62c5c" containerName="payload-pre-processing" containerID="cri-o://1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172" gracePeriod=30
Sep 11 14:13:57.887536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:57.887507    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:57.990323 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:57.990235    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:58.052014 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.051992    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:58.088485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.088451    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:58.161143 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.161117    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6hhq\" (UniqueName: \"kubernetes.io/projected/761e0ee4-031e-45a9-a834-da73e4d62c5c-kube-api-access-x6hhq\") pod \"761e0ee4-031e-45a9-a834-da73e4d62c5c\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") "
Sep 11 14:13:58.161301 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.161176    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761e0ee4-031e-45a9-a834-da73e4d62c5c-plugins-config-volume\") pod \"761e0ee4-031e-45a9-a834-da73e4d62c5c\" (UID: \"761e0ee4-031e-45a9-a834-da73e4d62c5c\") "
Sep 11 14:13:58.161542 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.161521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761e0ee4-031e-45a9-a834-da73e4d62c5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "761e0ee4-031e-45a9-a834-da73e4d62c5c" (UID: "761e0ee4-031e-45a9-a834-da73e4d62c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:13:58.163270 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.163251    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761e0ee4-031e-45a9-a834-da73e4d62c5c-kube-api-access-x6hhq" (OuterVolumeSpecName: "kube-api-access-x6hhq") pod "761e0ee4-031e-45a9-a834-da73e4d62c5c" (UID: "761e0ee4-031e-45a9-a834-da73e4d62c5c"). InnerVolumeSpecName "kube-api-access-x6hhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:13:58.233115 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.233078    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:58.262381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.262313    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6hhq\" (UniqueName: \"kubernetes.io/projected/761e0ee4-031e-45a9-a834-da73e4d62c5c-kube-api-access-x6hhq\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:13:58.262381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.262336    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761e0ee4-031e-45a9-a834-da73e4d62c5c-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:13:58.527661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.527601    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:58.560893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.560862    2558 generic.go:356] "Generic (PLEG): container finished" podID="761e0ee4-031e-45a9-a834-da73e4d62c5c" containerID="1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172" exitCode=0
Sep 11 14:13:58.561011 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.560905    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"
Sep 11 14:13:58.561011 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.560903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9" event={"ID":"761e0ee4-031e-45a9-a834-da73e4d62c5c","Type":"ContainerDied","Data":"1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172"}
Sep 11 14:13:58.561011 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.561005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9" event={"ID":"761e0ee4-031e-45a9-a834-da73e4d62c5c","Type":"ContainerDied","Data":"8a38dca6647d82b5034f237be4115fabd2787e8ed6f0e75dd8e0b55f8fa0f84e"}
Sep 11 14:13:58.561158 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.561053    2558 scope.go:117] "RemoveContainer" containerID="1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172"
Sep 11 14:13:58.575998 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.575961    2558 scope.go:117] "RemoveContainer" containerID="1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172"
Sep 11 14:13:58.576467 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:13:58.576442    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172\": container with ID starting with 1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172 not found: ID does not exist" containerID="1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172"
Sep 11 14:13:58.576555 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.576475    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172"} err="failed to get container status \"1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172\": rpc error: code = NotFound desc = could not find container \"1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172\": container with ID starting with 1f89e0156d63bbdcc3a73033375ef39d562f92663fe64a53d034e16ed76aa172 not found: ID does not exist"
Sep 11 14:13:58.582783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.582760    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"]
Sep 11 14:13:58.588040 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.588003    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dc719b7-9b6f9d95d-k7zv9"]
Sep 11 14:13:58.784126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.784043    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:58.869952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:58.869918    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:59.373500 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:59.373467    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:59.761113 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:59.761080    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:13:59.930133 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:13:59.930093    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:00.407399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:00.407369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:00.503372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:00.503339    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761e0ee4-031e-45a9-a834-da73e4d62c5c" path="/var/lib/kubelet/pods/761e0ee4-031e-45a9-a834-da73e4d62c5c/volumes"
Sep 11 14:14:02.980163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:02.980130    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:03.486295 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:03.486256    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:03.654387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:03.654356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:04.803317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:04.803280    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:06.017531 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:06.017493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:06.906426 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:06.906391    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:14:07.113482 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.113448    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"]
Sep 11 14:14:07.113970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.113952    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761e0ee4-031e-45a9-a834-da73e4d62c5c" containerName="payload-pre-processing"
Sep 11 14:14:07.114074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.113973    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="761e0ee4-031e-45a9-a834-da73e4d62c5c" containerName="payload-pre-processing"
Sep 11 14:14:07.114134 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.114091    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="761e0ee4-031e-45a9-a834-da73e4d62c5c" containerName="payload-pre-processing"
Sep 11 14:14:07.119990 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.119968    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.124316 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.124297    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-59484b3d-serving-cert\""
Sep 11 14:14:07.135021 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.134999    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"]
Sep 11 14:14:07.193681 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.193650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:07.241528 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.241506    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhm5\" (UniqueName: \"kubernetes.io/projected/3fabcab8-81ec-4353-ac72-e252352b84b5-kube-api-access-xbhm5\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.241608 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.241549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcab8-81ec-4353-ac72-e252352b84b5-openshift-service-ca\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.241608 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.241572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.241694 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.241620    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-metrics-certs\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.342913 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.342885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhm5\" (UniqueName: \"kubernetes.io/projected/3fabcab8-81ec-4353-ac72-e252352b84b5-kube-api-access-xbhm5\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.343056 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.342931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcab8-81ec-4353-ac72-e252352b84b5-openshift-service-ca\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.343056 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.342953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.343056 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.342982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-metrics-certs\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.343167 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:14:07.343114    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-serving-cert: secret "maas-api-e2e-migrate-59484b3d-serving-cert" not found
Sep 11 14:14:07.343200 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:14:07.343189    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls podName:3fabcab8-81ec-4353-ac72-e252352b84b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:07.843170923 +0000 UTC m=+981.806785880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls") pod "maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" (UID: "3fabcab8-81ec-4353-ac72-e252352b84b5") : secret "maas-api-e2e-migrate-59484b3d-serving-cert" not found
Sep 11 14:14:07.343540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.343518    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcab8-81ec-4353-ac72-e252352b84b5-openshift-service-ca\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.345345 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.345323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-metrics-certs\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.358917 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.358896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhm5\" (UniqueName: \"kubernetes.io/projected/3fabcab8-81ec-4353-ac72-e252352b84b5-kube-api-access-xbhm5\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.493390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.493314    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:07.739445 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.739411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:07.847546 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.847468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:07.849694 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:07.849675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls\") pod \"maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:08.029957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.029923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:08.037381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.037349    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:08.192419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.192391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:08.309757 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.309730    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"]
Sep 11 14:14:08.312746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.312726    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.317978 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.317959    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-59484b3d-dockercfg-g6cqt\""
Sep 11 14:14:08.323912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.323891    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"]
Sep 11 14:14:08.327099 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:14:08.327075    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fabcab8_81ec_4353_ac72_e252352b84b5.slice/crio-0dbab8464cd2b470095b146df54af410f6b87271fa0bee152ad32021cf4d9217 WatchSource:0}: Error finding container 0dbab8464cd2b470095b146df54af410f6b87271fa0bee152ad32021cf4d9217: Status 404 returned error can't find the container with id 0dbab8464cd2b470095b146df54af410f6b87271fa0bee152ad32021cf4d9217
Sep 11 14:14:08.338306 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.338292    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-59484b3d\""
Sep 11 14:14:08.426397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.426369    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"]
Sep 11 14:14:08.451643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.451569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed6f21a-5fdf-4a72-b940-ed4209580379-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.451643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.451605    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghl4\" (UniqueName: \"kubernetes.io/projected/eed6f21a-5fdf-4a72-b940-ed4209580379-kube-api-access-tghl4\") pod \"payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.552833 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.552804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed6f21a-5fdf-4a72-b940-ed4209580379-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.552833 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.552834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghl4\" (UniqueName: \"kubernetes.io/projected/eed6f21a-5fdf-4a72-b940-ed4209580379-kube-api-access-tghl4\") pod \"payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.553471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.553451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed6f21a-5fdf-4a72-b940-ed4209580379-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.564516 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.564494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghl4\" (UniqueName: \"kubernetes.io/projected/eed6f21a-5fdf-4a72-b940-ed4209580379-kube-api-access-tghl4\") pod \"payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.608083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.608057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" event={"ID":"3fabcab8-81ec-4353-ac72-e252352b84b5","Type":"ContainerStarted","Data":"0dbab8464cd2b470095b146df54af410f6b87271fa0bee152ad32021cf4d9217"}
Sep 11 14:14:08.623205 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.623183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:08.878586 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.878509    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:08.901715 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:08.901696    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"]
Sep 11 14:14:08.904336 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:14:08.904304    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed6f21a_5fdf_4a72_b940_ed4209580379.slice/crio-180d93f44e633335d0b8c71aa85bf6886ddb99397f5e18f9852422823ed884e5 WatchSource:0}: Error finding container 180d93f44e633335d0b8c71aa85bf6886ddb99397f5e18f9852422823ed884e5: Status 404 returned error can't find the container with id 180d93f44e633335d0b8c71aa85bf6886ddb99397f5e18f9852422823ed884e5
Sep 11 14:14:09.614522 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:09.614491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b" event={"ID":"eed6f21a-5fdf-4a72-b940-ed4209580379","Type":"ContainerStarted","Data":"aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a"}
Sep 11 14:14:09.614941 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:09.614539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b" event={"ID":"eed6f21a-5fdf-4a72-b940-ed4209580379","Type":"ContainerStarted","Data":"180d93f44e633335d0b8c71aa85bf6886ddb99397f5e18f9852422823ed884e5"}
Sep 11 14:14:09.614941 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:09.614590    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:09.616081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:09.616059    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" event={"ID":"3fabcab8-81ec-4353-ac72-e252352b84b5","Type":"ContainerStarted","Data":"8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4"}
Sep 11 14:14:09.616200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:09.616186    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:09.651112 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:09.651055    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b" podStartSLOduration=1.6510124560000001 podStartE2EDuration="1.651012456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:09.650594565 +0000 UTC m=+983.614209550" watchObservedRunningTime="2026-09-11 14:14:09.651012456 +0000 UTC m=+983.614627434"
Sep 11 14:14:10.037399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:10.037375    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:11.057302 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.057253    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" podStartSLOduration=3.706090075 podStartE2EDuration="4.057237392s" totalImagesPullingTime="351.147317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:09.694764806 +0000 UTC m=+983.658379780" watchObservedRunningTime="2026-09-11 14:14:11.057237392 +0000 UTC m=+985.020852367"
Sep 11 14:14:11.057746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.057650    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"]
Sep 11 14:14:11.059957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.059942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.062452 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.062435    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c906f9bb-kserve-self-signed-certs\""
Sep 11 14:14:11.073581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.073560    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"]
Sep 11 14:14:11.171167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.171135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-dshm\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.171167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.171171    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5x4w\" (UniqueName: \"kubernetes.io/projected/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-kube-api-access-d5x4w\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.171378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.171261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tmp-dir\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.171378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.171300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tls-certs\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.171378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.171362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-home\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.171378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.171378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-model-cache\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tmp-dir\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tls-certs\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-home\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272571    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-model-cache\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-dshm\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5x4w\" (UniqueName: \"kubernetes.io/projected/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-kube-api-access-d5x4w\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.272953 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tmp-dir\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.273020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272964    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-home\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.273020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.272996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-model-cache\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.274844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.274819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-dshm\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.275224 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.275204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tls-certs\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.284985 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.284954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5x4w\" (UniqueName: \"kubernetes.io/projected/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-kube-api-access-d5x4w\") pod \"e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") " pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.369126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.369042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:11.390020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.389988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:11.639007 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:11.638983    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"]
Sep 11 14:14:11.641212 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:14:11.641186    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f4a9ef1_773e_4202_99e3_0f4a10f7933d.slice/crio-95d21fcf5afc445f48c16320a0d89fd3a9b959744c9868c281313e9653d1554e WatchSource:0}: Error finding container 95d21fcf5afc445f48c16320a0d89fd3a9b959744c9868c281313e9653d1554e: Status 404 returned error can't find the container with id 95d21fcf5afc445f48c16320a0d89fd3a9b959744c9868c281313e9653d1554e
Sep 11 14:14:12.631776 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:12.631742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff" event={"ID":"2f4a9ef1-773e-4202-99e3-0f4a10f7933d","Type":"ContainerStarted","Data":"0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166"}
Sep 11 14:14:12.632159 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:12.631783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff" event={"ID":"2f4a9ef1-773e-4202-99e3-0f4a10f7933d","Type":"ContainerStarted","Data":"95d21fcf5afc445f48c16320a0d89fd3a9b959744c9868c281313e9653d1554e"}
Sep 11 14:14:12.656310 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:12.656264    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff" podStartSLOduration=1.6562494490000002 podStartE2EDuration="1.656249449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:12.654513244 +0000 UTC m=+986.618128218" watchObservedRunningTime="2026-09-11 14:14:12.656249449 +0000 UTC m=+986.619864458"
Sep 11 14:14:15.625823 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:15.625796    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:20.623086 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:20.623059    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:21.369589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:21.369553    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:21.369589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:21.369597    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:21.381886 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:21.381862    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:21.682268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:21.682183    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:14:24.977086 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:24.977056    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:14:28.086992 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:28.086960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:30.227344 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:30.227309    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:31.186477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:31.186441    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:31.996316 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:31.996281    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:32.555094 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:32.555046    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:33.349222 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:33.349181    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:34.396775 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:34.396733    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:35.401354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:35.401321    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:36.455653 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:36.455617    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:37.531600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:37.531565    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:38.475240 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:38.475208    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:44.579679 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:44.579640    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:45.357155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.357120    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"]
Sep 11 14:14:45.357354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.357332    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" podUID="3fabcab8-81ec-4353-ac72-e252352b84b5" containerName="maas-api" containerID="cri-o://8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4" gracePeriod=30
Sep 11 14:14:45.425301 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.425280    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:45.515358 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.515324    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"]
Sep 11 14:14:45.515541 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.515516    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b" podUID="eed6f21a-5fdf-4a72-b940-ed4209580379" containerName="payload-pre-processing" containerID="cri-o://aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a" gracePeriod=30
Sep 11 14:14:45.580593 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.580566    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:45.640180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.640161    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:45.656749 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.656724    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbhm5\" (UniqueName: \"kubernetes.io/projected/3fabcab8-81ec-4353-ac72-e252352b84b5-kube-api-access-xbhm5\") pod \"3fabcab8-81ec-4353-ac72-e252352b84b5\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") "
Sep 11 14:14:45.656907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.656776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcab8-81ec-4353-ac72-e252352b84b5-openshift-service-ca\") pod \"3fabcab8-81ec-4353-ac72-e252352b84b5\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") "
Sep 11 14:14:45.656907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.656829    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls\") pod \"3fabcab8-81ec-4353-ac72-e252352b84b5\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") "
Sep 11 14:14:45.657037 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.656991    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-metrics-certs\") pod \"3fabcab8-81ec-4353-ac72-e252352b84b5\" (UID: \"3fabcab8-81ec-4353-ac72-e252352b84b5\") "
Sep 11 14:14:45.657151 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.657130    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fabcab8-81ec-4353-ac72-e252352b84b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3fabcab8-81ec-4353-ac72-e252352b84b5" (UID: "3fabcab8-81ec-4353-ac72-e252352b84b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:14:45.657329 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.657309    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabcab8-81ec-4353-ac72-e252352b84b5-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:14:45.659172 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.659148    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3fabcab8-81ec-4353-ac72-e252352b84b5" (UID: "3fabcab8-81ec-4353-ac72-e252352b84b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:14:45.659296 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.659275    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3fabcab8-81ec-4353-ac72-e252352b84b5" (UID: "3fabcab8-81ec-4353-ac72-e252352b84b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:14:45.659351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.659296    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fabcab8-81ec-4353-ac72-e252352b84b5-kube-api-access-xbhm5" (OuterVolumeSpecName: "kube-api-access-xbhm5") pod "3fabcab8-81ec-4353-ac72-e252352b84b5" (UID: "3fabcab8-81ec-4353-ac72-e252352b84b5"). InnerVolumeSpecName "kube-api-access-xbhm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:14:45.739754 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.739717    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:45.758438 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.758413    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:45.758555 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.758424    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:14:45.758555 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.758518    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fabcab8-81ec-4353-ac72-e252352b84b5-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:14:45.758555 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.758528    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhm5\" (UniqueName: \"kubernetes.io/projected/3fabcab8-81ec-4353-ac72-e252352b84b5-kube-api-access-xbhm5\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:14:45.774812 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.774784    2558 generic.go:356] "Generic (PLEG): container finished" podID="eed6f21a-5fdf-4a72-b940-ed4209580379" containerID="aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a" exitCode=0
Sep 11 14:14:45.774935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.774814    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"
Sep 11 14:14:45.774935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.774871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b" event={"ID":"eed6f21a-5fdf-4a72-b940-ed4209580379","Type":"ContainerDied","Data":"aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a"}
Sep 11 14:14:45.774935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.774916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b" event={"ID":"eed6f21a-5fdf-4a72-b940-ed4209580379","Type":"ContainerDied","Data":"180d93f44e633335d0b8c71aa85bf6886ddb99397f5e18f9852422823ed884e5"}
Sep 11 14:14:45.775075 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.774938    2558 scope.go:117] "RemoveContainer" containerID="aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a"
Sep 11 14:14:45.776112 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.776093    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fabcab8-81ec-4353-ac72-e252352b84b5" containerID="8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4" exitCode=0
Sep 11 14:14:45.776181 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.776123    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"
Sep 11 14:14:45.776225 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.776123    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" event={"ID":"3fabcab8-81ec-4353-ac72-e252352b84b5","Type":"ContainerDied","Data":"8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4"}
Sep 11 14:14:45.776271 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.776229    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" event={"ID":"3fabcab8-81ec-4353-ac72-e252352b84b5","Type":"ContainerDied","Data":"0dbab8464cd2b470095b146df54af410f6b87271fa0bee152ad32021cf4d9217"}
Sep 11 14:14:45.786320 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.786099    2558 scope.go:117] "RemoveContainer" containerID="aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a"
Sep 11 14:14:45.786926 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:14:45.786904    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a\": container with ID starting with aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a not found: ID does not exist" containerID="aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a"
Sep 11 14:14:45.787051 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.786970    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a"} err="failed to get container status \"aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a\": rpc error: code = NotFound desc = could not find container \"aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a\": container with ID starting with aa3568c0545649cddbcf3592447d110e40c48094f87ac0ee93be1d8a7391f47a not found: ID does not exist"
Sep 11 14:14:45.787051 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.786994    2558 scope.go:117] "RemoveContainer" containerID="8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4"
Sep 11 14:14:45.796313 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.796286    2558 scope.go:117] "RemoveContainer" containerID="8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4"
Sep 11 14:14:45.796591 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:14:45.796574    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4\": container with ID starting with 8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4 not found: ID does not exist" containerID="8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4"
Sep 11 14:14:45.796659 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.796596    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4"} err="failed to get container status \"8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4\": rpc error: code = NotFound desc = could not find container \"8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4\": container with ID starting with 8018c73a4679a21d588368f9151b749b1f50f4172d7280c8358df673f1c2f5a4 not found: ID does not exist"
Sep 11 14:14:45.817577 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.817551    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"]
Sep 11 14:14:45.821897 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.821875    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj"]
Sep 11 14:14:45.856665 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.856642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:45.859718 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.859700    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed6f21a-5fdf-4a72-b940-ed4209580379-plugins-config-volume\") pod \"eed6f21a-5fdf-4a72-b940-ed4209580379\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") "
Sep 11 14:14:45.859798 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.859728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tghl4\" (UniqueName: \"kubernetes.io/projected/eed6f21a-5fdf-4a72-b940-ed4209580379-kube-api-access-tghl4\") pod \"eed6f21a-5fdf-4a72-b940-ed4209580379\" (UID: \"eed6f21a-5fdf-4a72-b940-ed4209580379\") "
Sep 11 14:14:45.860103 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.860083    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed6f21a-5fdf-4a72-b940-ed4209580379-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eed6f21a-5fdf-4a72-b940-ed4209580379" (UID: "eed6f21a-5fdf-4a72-b940-ed4209580379"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:14:45.861878 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.861828    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed6f21a-5fdf-4a72-b940-ed4209580379-kube-api-access-tghl4" (OuterVolumeSpecName: "kube-api-access-tghl4") pod "eed6f21a-5fdf-4a72-b940-ed4209580379" (UID: "eed6f21a-5fdf-4a72-b940-ed4209580379"). InnerVolumeSpecName "kube-api-access-tghl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:14:45.960983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.960950    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed6f21a-5fdf-4a72-b940-ed4209580379-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:14:45.960983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:45.960979    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghl4\" (UniqueName: \"kubernetes.io/projected/eed6f21a-5fdf-4a72-b940-ed4209580379-kube-api-access-tghl4\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:14:46.020187 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.020153    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:46.098599 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.098562    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"]
Sep 11 14:14:46.105675 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.105649    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59484b3d-866f76dc46-dj89b"]
Sep 11 14:14:46.504605 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.504579    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fabcab8-81ec-4353-ac72-e252352b84b5" path="/var/lib/kubelet/pods/3fabcab8-81ec-4353-ac72-e252352b84b5/volumes"
Sep 11 14:14:46.504918 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.504907    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed6f21a-5fdf-4a72-b940-ed4209580379" path="/var/lib/kubelet/pods/eed6f21a-5fdf-4a72-b940-ed4209580379/volumes"
Sep 11 14:14:46.706802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.706770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:46.905354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:46.905329    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:47.175654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:47.175580    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:47.723247 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:47.723219    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:48.253265 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:48.253233    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:48.621188 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:48.621086    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59484b3d-649bc577fd-rbrmj" podUID="3fabcab8-81ec-4353-ac72-e252352b84b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 14:14:49.362925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:49.362889    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:49.988196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:49.988161    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:50.103080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:50.103056    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:50.486076 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:50.486041    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:51.379670 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.379643    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"]
Sep 11 14:14:51.380008 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.379994    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fabcab8-81ec-4353-ac72-e252352b84b5" containerName="maas-api"
Sep 11 14:14:51.380124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.380010    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fabcab8-81ec-4353-ac72-e252352b84b5" containerName="maas-api"
Sep 11 14:14:51.380124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.380060    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed6f21a-5fdf-4a72-b940-ed4209580379" containerName="payload-pre-processing"
Sep 11 14:14:51.380124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.380066    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed6f21a-5fdf-4a72-b940-ed4209580379" containerName="payload-pre-processing"
Sep 11 14:14:51.380267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.380135    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed6f21a-5fdf-4a72-b940-ed4209580379" containerName="payload-pre-processing"
Sep 11 14:14:51.380267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.380145    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fabcab8-81ec-4353-ac72-e252352b84b5" containerName="maas-api"
Sep 11 14:14:51.390947 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.390920    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.393639 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.393611    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-65bba332-serving-cert\""
Sep 11 14:14:51.401075 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.401049    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"]
Sep 11 14:14:51.448758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.448731    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:51.501451 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.501428    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbf6\" (UniqueName: \"kubernetes.io/projected/55c87497-5cf6-4899-afd2-0ec8361ac0f5-kube-api-access-vwbf6\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.501727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.501457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c87497-5cf6-4899-afd2-0ec8361ac0f5-openshift-service-ca\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.501727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.501492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.501727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.501564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-metrics-certs\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.546133 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.546109    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:51.602721 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.602697    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbf6\" (UniqueName: \"kubernetes.io/projected/55c87497-5cf6-4899-afd2-0ec8361ac0f5-kube-api-access-vwbf6\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.602823 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.602732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c87497-5cf6-4899-afd2-0ec8361ac0f5-openshift-service-ca\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.602883 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.602864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.602969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.602949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-metrics-certs\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.603058 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:14:51.602971    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-serving-cert: secret "maas-api-e2e-ait-65bba332-serving-cert" not found
Sep 11 14:14:51.603126 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:14:51.603064    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls podName:55c87497-5cf6-4899-afd2-0ec8361ac0f5 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:52.103045766 +0000 UTC m=+1026.066660734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls") pod "maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" (UID: "55c87497-5cf6-4899-afd2-0ec8361ac0f5") : secret "maas-api-e2e-ait-65bba332-serving-cert" not found
Sep 11 14:14:51.603437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.603417    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c87497-5cf6-4899-afd2-0ec8361ac0f5-openshift-service-ca\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.605237 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.605218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-metrics-certs\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.617760 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.617735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbf6\" (UniqueName: \"kubernetes.io/projected/55c87497-5cf6-4899-afd2-0ec8361ac0f5-kube-api-access-vwbf6\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:51.959375 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:51.959344    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:52.073467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.073436    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:14:52.092480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.092448    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:52.108144 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.108114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:52.111086 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.111056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls\") pod \"maas-api-e2e-ait-65bba332-64f68946f4-wlvx5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:52.192914 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.192888    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:52.288070 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.288002    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"]
Sep 11 14:14:52.291667 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.291652    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.294618 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.294595    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-65bba332\""
Sep 11 14:14:52.294618 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.294595    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-65bba332-dockercfg-5gqxn\""
Sep 11 14:14:52.302199 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.302180    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"]
Sep 11 14:14:52.304603 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.304583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:52.410844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.410808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64da0985-607c-417f-a1f0-37a6c1866380-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.410958 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.410886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbqh\" (UniqueName: \"kubernetes.io/projected/64da0985-607c-417f-a1f0-37a6c1866380-kube-api-access-8zbqh\") pod \"payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.511988 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.511963    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbqh\" (UniqueName: \"kubernetes.io/projected/64da0985-607c-417f-a1f0-37a6c1866380-kube-api-access-8zbqh\") pod \"payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.512292 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.512023    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64da0985-607c-417f-a1f0-37a6c1866380-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.512629 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.512611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64da0985-607c-417f-a1f0-37a6c1866380-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.527583 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.527563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbqh\" (UniqueName: \"kubernetes.io/projected/64da0985-607c-417f-a1f0-37a6c1866380-kube-api-access-8zbqh\") pod \"payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.567927 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.567875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:52.582388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.582370    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"]
Sep 11 14:14:52.584399 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:14:52.584379    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c87497_5cf6_4899_afd2_0ec8361ac0f5.slice/crio-0aded2c0d082c64d8a18db152e54d9bf99737caacec2f1bb085635a7222c9460 WatchSource:0}: Error finding container 0aded2c0d082c64d8a18db152e54d9bf99737caacec2f1bb085635a7222c9460: Status 404 returned error can't find the container with id 0aded2c0d082c64d8a18db152e54d9bf99737caacec2f1bb085635a7222c9460
Sep 11 14:14:52.602315 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.602296    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:52.807370 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.807341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" event={"ID":"55c87497-5cf6-4899-afd2-0ec8361ac0f5","Type":"ContainerStarted","Data":"0aded2c0d082c64d8a18db152e54d9bf99737caacec2f1bb085635a7222c9460"}
Sep 11 14:14:52.871254 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:52.871234    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"]
Sep 11 14:14:52.872573 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:14:52.872542    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64da0985_607c_417f_a1f0_37a6c1866380.slice/crio-bb3031dea17183f6cac94dee8f9fc31c98228b5ef957c63059df90946bfac168 WatchSource:0}: Error finding container bb3031dea17183f6cac94dee8f9fc31c98228b5ef957c63059df90946bfac168: Status 404 returned error can't find the container with id bb3031dea17183f6cac94dee8f9fc31c98228b5ef957c63059df90946bfac168
Sep 11 14:14:53.474917 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.474888    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:53.813865 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.813789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" event={"ID":"55c87497-5cf6-4899-afd2-0ec8361ac0f5","Type":"ContainerStarted","Data":"99ef5c05596073bcfc8f2fa58b95b1a58851e8092272fb4362bf48582cdcffce"}
Sep 11 14:14:53.814431 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.813937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:14:53.815439 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.815415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn" event={"ID":"64da0985-607c-417f-a1f0-37a6c1866380","Type":"ContainerStarted","Data":"e03b66f470df6e0b859a87df9b2f0ac92045b4ccbd69c6f07e53bb33bf34a4ec"}
Sep 11 14:14:53.815572 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.815456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn" event={"ID":"64da0985-607c-417f-a1f0-37a6c1866380","Type":"ContainerStarted","Data":"bb3031dea17183f6cac94dee8f9fc31c98228b5ef957c63059df90946bfac168"}
Sep 11 14:14:53.815572 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.815518    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:14:53.837104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.837066    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" podStartSLOduration=2.382296234 podStartE2EDuration="2.837052142s" totalImagesPullingTime="454.755908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:53.836135468 +0000 UTC m=+1027.799750450" watchObservedRunningTime="2026-09-11 14:14:53.837052142 +0000 UTC m=+1027.800667111"
Sep 11 14:14:53.859420 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:53.859375    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn" podStartSLOduration=1.859359319 podStartE2EDuration="1.859359319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:53.857079301 +0000 UTC m=+1027.820694276" watchObservedRunningTime="2026-09-11 14:14:53.859359319 +0000 UTC m=+1027.822974294"
Sep 11 14:14:54.331815 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:54.331786    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:55.492963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:55.492927    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:14:59.825077 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:14:59.825047    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:15:04.821195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:04.821164    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:15:07.583007 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:07.582976    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:15:09.257832 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:09.257802    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:12.554407 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:12.554371    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:14.161222 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:14.161188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:14.968094 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:14.968051    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:16.065910 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:16.065866    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:17.153284 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:17.153251    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:18.278918 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:18.278887    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:19.169677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:19.169641    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:20.001087 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:20.001049    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:20.979869 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:20.979834    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:21.475243 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.475212    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:15:21.682826 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.682795    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"]
Sep 11 14:15:21.683085 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.683062    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" podUID="55c87497-5cf6-4899-afd2-0ec8361ac0f5" containerName="maas-api" containerID="cri-o://99ef5c05596073bcfc8f2fa58b95b1a58851e8092272fb4362bf48582cdcffce" gracePeriod=30
Sep 11 14:15:21.732136 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.732061    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:21.794504 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.794466    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"]
Sep 11 14:15:21.794740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.794694    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn" podUID="64da0985-607c-417f-a1f0-37a6c1866380" containerName="payload-pre-processing" containerID="cri-o://e03b66f470df6e0b859a87df9b2f0ac92045b4ccbd69c6f07e53bb33bf34a4ec" gracePeriod=30
Sep 11 14:15:21.940498 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.940452    2558 generic.go:356] "Generic (PLEG): container finished" podID="64da0985-607c-417f-a1f0-37a6c1866380" containerID="e03b66f470df6e0b859a87df9b2f0ac92045b4ccbd69c6f07e53bb33bf34a4ec" exitCode=0
Sep 11 14:15:21.940746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.940680    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn" event={"ID":"64da0985-607c-417f-a1f0-37a6c1866380","Type":"ContainerDied","Data":"e03b66f470df6e0b859a87df9b2f0ac92045b4ccbd69c6f07e53bb33bf34a4ec"}
Sep 11 14:15:21.942232 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.942211    2558 generic.go:356] "Generic (PLEG): container finished" podID="55c87497-5cf6-4899-afd2-0ec8361ac0f5" containerID="99ef5c05596073bcfc8f2fa58b95b1a58851e8092272fb4362bf48582cdcffce" exitCode=0
Sep 11 14:15:21.942317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.942255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" event={"ID":"55c87497-5cf6-4899-afd2-0ec8361ac0f5","Type":"ContainerDied","Data":"99ef5c05596073bcfc8f2fa58b95b1a58851e8092272fb4362bf48582cdcffce"}
Sep 11 14:15:21.942317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.942297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5" event={"ID":"55c87497-5cf6-4899-afd2-0ec8361ac0f5","Type":"ContainerDied","Data":"0aded2c0d082c64d8a18db152e54d9bf99737caacec2f1bb085635a7222c9460"}
Sep 11 14:15:21.942317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.942313    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aded2c0d082c64d8a18db152e54d9bf99737caacec2f1bb085635a7222c9460"
Sep 11 14:15:21.956358 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:21.956338    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:15:22.045651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.045622    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwbf6\" (UniqueName: \"kubernetes.io/projected/55c87497-5cf6-4899-afd2-0ec8361ac0f5-kube-api-access-vwbf6\") pod \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") "
Sep 11 14:15:22.045793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.045670    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c87497-5cf6-4899-afd2-0ec8361ac0f5-openshift-service-ca\") pod \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") "
Sep 11 14:15:22.045793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.045690    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls\") pod \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") "
Sep 11 14:15:22.045793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.045714    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-metrics-certs\") pod \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\" (UID: \"55c87497-5cf6-4899-afd2-0ec8361ac0f5\") "
Sep 11 14:15:22.046060 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.046021    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c87497-5cf6-4899-afd2-0ec8361ac0f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55c87497-5cf6-4899-afd2-0ec8361ac0f5" (UID: "55c87497-5cf6-4899-afd2-0ec8361ac0f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:15:22.047756 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.047733    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55c87497-5cf6-4899-afd2-0ec8361ac0f5" (UID: "55c87497-5cf6-4899-afd2-0ec8361ac0f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:15:22.047889 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.047753    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c87497-5cf6-4899-afd2-0ec8361ac0f5-kube-api-access-vwbf6" (OuterVolumeSpecName: "kube-api-access-vwbf6") pod "55c87497-5cf6-4899-afd2-0ec8361ac0f5" (UID: "55c87497-5cf6-4899-afd2-0ec8361ac0f5"). InnerVolumeSpecName "kube-api-access-vwbf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:15:22.047889 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.047770    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55c87497-5cf6-4899-afd2-0ec8361ac0f5" (UID: "55c87497-5cf6-4899-afd2-0ec8361ac0f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:15:22.057808 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.057790    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:15:22.147022 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.146992    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zbqh\" (UniqueName: \"kubernetes.io/projected/64da0985-607c-417f-a1f0-37a6c1866380-kube-api-access-8zbqh\") pod \"64da0985-607c-417f-a1f0-37a6c1866380\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") "
Sep 11 14:15:22.147218 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.147134    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64da0985-607c-417f-a1f0-37a6c1866380-plugins-config-volume\") pod \"64da0985-607c-417f-a1f0-37a6c1866380\" (UID: \"64da0985-607c-417f-a1f0-37a6c1866380\") "
Sep 11 14:15:22.147413 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.147393    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:15:22.147472 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.147417    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55c87497-5cf6-4899-afd2-0ec8361ac0f5-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:15:22.147472 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.147433    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbf6\" (UniqueName: \"kubernetes.io/projected/55c87497-5cf6-4899-afd2-0ec8361ac0f5-kube-api-access-vwbf6\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:15:22.147472 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.147447    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c87497-5cf6-4899-afd2-0ec8361ac0f5-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:15:22.147585 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.147511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64da0985-607c-417f-a1f0-37a6c1866380-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64da0985-607c-417f-a1f0-37a6c1866380" (UID: "64da0985-607c-417f-a1f0-37a6c1866380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:15:22.149007 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.148983    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64da0985-607c-417f-a1f0-37a6c1866380-kube-api-access-8zbqh" (OuterVolumeSpecName: "kube-api-access-8zbqh") pod "64da0985-607c-417f-a1f0-37a6c1866380" (UID: "64da0985-607c-417f-a1f0-37a6c1866380"). InnerVolumeSpecName "kube-api-access-8zbqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:15:22.159804 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.159778    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"]
Sep 11 14:15:22.160202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.160186    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c87497-5cf6-4899-afd2-0ec8361ac0f5" containerName="maas-api"
Sep 11 14:15:22.160280 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.160204    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c87497-5cf6-4899-afd2-0ec8361ac0f5" containerName="maas-api"
Sep 11 14:15:22.160280 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.160215    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64da0985-607c-417f-a1f0-37a6c1866380" containerName="payload-pre-processing"
Sep 11 14:15:22.160280 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.160224    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="64da0985-607c-417f-a1f0-37a6c1866380" containerName="payload-pre-processing"
Sep 11 14:15:22.160430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.160319    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c87497-5cf6-4899-afd2-0ec8361ac0f5" containerName="maas-api"
Sep 11 14:15:22.160430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.160335    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="64da0985-607c-417f-a1f0-37a6c1866380" containerName="payload-pre-processing"
Sep 11 14:15:22.163226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.163209    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.166341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.166323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2a21d9\"/\"autores-model-2a21d9-kserve-self-signed-certs\""
Sep 11 14:15:22.166455 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.166327    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2a21d9\"/\"kube-root-ca.crt\""
Sep 11 14:15:22.166570 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.166552    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2a21d9\"/\"default-dockercfg-ftdt7\""
Sep 11 14:15:22.166639 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.166621    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2a21d9\"/\"openshift-service-ca.crt\""
Sep 11 14:15:22.182719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.182696    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"]
Sep 11 14:15:22.247826 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.247801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1234bb00-6f5e-4022-bee9-aba3e05a624c-tls-certs\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.247933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.247852    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-dshm\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.247933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.247896    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-home\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.247933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.247928    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-model-cache\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.248052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.247951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-tmp-dir\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.248052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.247973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv54\" (UniqueName: \"kubernetes.io/projected/1234bb00-6f5e-4022-bee9-aba3e05a624c-kube-api-access-btv54\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.248052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.248017    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbqh\" (UniqueName: \"kubernetes.io/projected/64da0985-607c-417f-a1f0-37a6c1866380-kube-api-access-8zbqh\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:15:22.248052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.248047    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64da0985-607c-417f-a1f0-37a6c1866380-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:15:22.348771 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.348700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-home\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.348771 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.348747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-model-cache\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.348771 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.348763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-tmp-dir\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.348983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.348784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv54\" (UniqueName: \"kubernetes.io/projected/1234bb00-6f5e-4022-bee9-aba3e05a624c-kube-api-access-btv54\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.348983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.348825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1234bb00-6f5e-4022-bee9-aba3e05a624c-tls-certs\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.348983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.348893    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-dshm\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.349188 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.349166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-home\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.349188 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.349180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-model-cache\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.349268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.349228    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-tmp-dir\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.351064 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.351045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-dshm\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.351404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.351345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1234bb00-6f5e-4022-bee9-aba3e05a624c-tls-certs\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.358056 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.358023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv54\" (UniqueName: \"kubernetes.io/projected/1234bb00-6f5e-4022-bee9-aba3e05a624c-kube-api-access-btv54\") pod \"autores-model-2a21d9-kserve-8bb7679f9-42vlv\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") " pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.471909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.471887    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:22.585687 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.585658    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:22.692670 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.692631    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:22.757727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.757700    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"]
Sep 11 14:15:22.759720 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:15:22.759693    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1234bb00_6f5e_4022_bee9_aba3e05a624c.slice/crio-ef31f939dcca1a5a7b1f57c325e8c435e40188097ab57f172806dc6363c64957 WatchSource:0}: Error finding container ef31f939dcca1a5a7b1f57c325e8c435e40188097ab57f172806dc6363c64957: Status 404 returned error can't find the container with id ef31f939dcca1a5a7b1f57c325e8c435e40188097ab57f172806dc6363c64957
Sep 11 14:15:22.834634 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.834605    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:22.951238 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.951144    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv" event={"ID":"1234bb00-6f5e-4022-bee9-aba3e05a624c","Type":"ContainerStarted","Data":"1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3"}
Sep 11 14:15:22.951238 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.951190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv" event={"ID":"1234bb00-6f5e-4022-bee9-aba3e05a624c","Type":"ContainerStarted","Data":"ef31f939dcca1a5a7b1f57c325e8c435e40188097ab57f172806dc6363c64957"}
Sep 11 14:15:22.952575 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.952549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn" event={"ID":"64da0985-607c-417f-a1f0-37a6c1866380","Type":"ContainerDied","Data":"bb3031dea17183f6cac94dee8f9fc31c98228b5ef957c63059df90946bfac168"}
Sep 11 14:15:22.952692 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.952605    2558 scope.go:117] "RemoveContainer" containerID="e03b66f470df6e0b859a87df9b2f0ac92045b4ccbd69c6f07e53bb33bf34a4ec"
Sep 11 14:15:22.952692 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.952538    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"
Sep 11 14:15:22.952692 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.952558    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"
Sep 11 14:15:22.974697 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.974656    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv" podStartSLOduration=0.974642986 podStartE2EDuration="974.642986ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:22.974406925 +0000 UTC m=+1056.938021899" watchObservedRunningTime="2026-09-11 14:15:22.974642986 +0000 UTC m=+1056.938257959"
Sep 11 14:15:22.994412 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:22.994387    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"]
Sep 11 14:15:23.005425 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:23.005406    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65bba332-7bf9594c66-c8czn"]
Sep 11 14:15:23.016395 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:23.016374    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"]
Sep 11 14:15:23.019999 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:23.019979    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-65bba332-64f68946f4-wlvx5"]
Sep 11 14:15:23.345948 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:23.345861    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:23.686237 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:23.686166    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:23.792253 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:23.792225    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:24.104767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:24.104731    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:24.197584 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:24.197538    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:24.303653 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:24.303614    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:24.508543 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:24.508511    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c87497-5cf6-4899-afd2-0ec8361ac0f5" path="/var/lib/kubelet/pods/55c87497-5cf6-4899-afd2-0ec8361ac0f5/volumes"
Sep 11 14:15:24.508853 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:24.508841    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64da0985-607c-417f-a1f0-37a6c1866380" path="/var/lib/kubelet/pods/64da0985-607c-417f-a1f0-37a6c1866380/volumes"
Sep 11 14:15:24.826273 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:24.826189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:26.081698 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:26.081649    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:26.424804 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:26.424725    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:26.621149 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:26.621118    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:27.650819 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:27.650778    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:27.763467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:27.763435    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:28.530409 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.530378    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:28.551158 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.551130    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"]
Sep 11 14:15:28.555704 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.555689    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.558871 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.558853    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d609a19a-dockercfg-hjxxn\""
Sep 11 14:15:28.558964 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.558914    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d609a19a\""
Sep 11 14:15:28.568618 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.568587    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"]
Sep 11 14:15:28.611930 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.611900    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"]
Sep 11 14:15:28.615023 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.615009    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.632450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.632425    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"]
Sep 11 14:15:28.646289 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.646266    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:28.703005 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.702980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266w6\" (UniqueName: \"kubernetes.io/projected/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-kube-api-access-266w6\") pod \"payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.703322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.703052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25eac3ee-1027-496b-9a35-92cf639cb94a-plugins-config-volume\") pod \"payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") " pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.703322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.703096    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876kx\" (UniqueName: \"kubernetes.io/projected/25eac3ee-1027-496b-9a35-92cf639cb94a-kube-api-access-876kx\") pod \"payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") " pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.703322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.703120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.803766 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.803712    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876kx\" (UniqueName: \"kubernetes.io/projected/25eac3ee-1027-496b-9a35-92cf639cb94a-kube-api-access-876kx\") pod \"payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") " pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.803766 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.803742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.803936 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.803777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266w6\" (UniqueName: \"kubernetes.io/projected/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-kube-api-access-266w6\") pod \"payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.803936 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.803829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25eac3ee-1027-496b-9a35-92cf639cb94a-plugins-config-volume\") pod \"payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") " pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.804404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.804386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.804491 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.804474    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25eac3ee-1027-496b-9a35-92cf639cb94a-plugins-config-volume\") pod \"payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") " pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.817651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.817632    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-876kx\" (UniqueName: \"kubernetes.io/projected/25eac3ee-1027-496b-9a35-92cf639cb94a-kube-api-access-876kx\") pod \"payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") " pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.819180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.819160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-266w6\" (UniqueName: \"kubernetes.io/projected/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-kube-api-access-266w6\") pod \"payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.864825 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.864805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:28.924673 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.924640    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:28.989441 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:28.989391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:29.150566 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.150540    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"]
Sep 11 14:15:29.152116 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:15:29.152096    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc8dc58_2e35_4327_8570_b6b8963bc6b2.slice/crio-44bba5f8638b282ff3aa24632710329d88666684dd87d4569be4f1e0a7f75c03 WatchSource:0}: Error finding container 44bba5f8638b282ff3aa24632710329d88666684dd87d4569be4f1e0a7f75c03: Status 404 returned error can't find the container with id 44bba5f8638b282ff3aa24632710329d88666684dd87d4569be4f1e0a7f75c03
Sep 11 14:15:29.210576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.210558    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"]
Sep 11 14:15:29.212152 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:15:29.212129    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25eac3ee_1027_496b_9a35_92cf639cb94a.slice/crio-eab23ff69cf51e7506426e4f39f0a51ed046868f7a3e541bf546e721f1a836f7 WatchSource:0}: Error finding container eab23ff69cf51e7506426e4f39f0a51ed046868f7a3e541bf546e721f1a836f7: Status 404 returned error can't find the container with id eab23ff69cf51e7506426e4f39f0a51ed046868f7a3e541bf546e721f1a836f7
Sep 11 14:15:29.992663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.992623    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m" event={"ID":"25eac3ee-1027-496b-9a35-92cf639cb94a","Type":"ContainerStarted","Data":"69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902"}
Sep 11 14:15:29.993153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.992728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m" event={"ID":"25eac3ee-1027-496b-9a35-92cf639cb94a","Type":"ContainerStarted","Data":"eab23ff69cf51e7506426e4f39f0a51ed046868f7a3e541bf546e721f1a836f7"}
Sep 11 14:15:29.993153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.992957    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:29.994267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.994243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs" event={"ID":"1fc8dc58-2e35-4327-8570-b6b8963bc6b2","Type":"ContainerStarted","Data":"eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076"}
Sep 11 14:15:29.994404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.994284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs" event={"ID":"1fc8dc58-2e35-4327-8570-b6b8963bc6b2","Type":"ContainerStarted","Data":"44bba5f8638b282ff3aa24632710329d88666684dd87d4569be4f1e0a7f75c03"}
Sep 11 14:15:29.994404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:29.994335    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:30.014421 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:30.014384    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m" podStartSLOduration=2.014371747 podStartE2EDuration="2.014371747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:30.012064651 +0000 UTC m=+1063.975679621" watchObservedRunningTime="2026-09-11 14:15:30.014371747 +0000 UTC m=+1063.977986720"
Sep 11 14:15:30.039309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:30.039271    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs" podStartSLOduration=2.039258251 podStartE2EDuration="2.039258251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:30.036522373 +0000 UTC m=+1064.000137345" watchObservedRunningTime="2026-09-11 14:15:30.039258251 +0000 UTC m=+1064.002873224"
Sep 11 14:15:30.210334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:30.210291    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:31.469348 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:31.469305    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:32.355621 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:32.355585    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:32.472486 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:32.472462    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:32.472796 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:32.472537    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:32.484471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:32.484448    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:33.020555 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:33.020524    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:15:38.680020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:38.679986    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:15:41.001524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:41.001492    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:41.001897 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:41.001671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:50.434858 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:50.434821    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:51.522609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:51.522571    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:52.880200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:52.880169    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:53.862536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:53.862502    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:55.147000 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:55.146965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:56.065862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:56.065822    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:57.085400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:57.085367    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:57.886520 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:57.886486    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:58.795772 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:58.795727    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:59.542445 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.542414    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:59.605571 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.605538    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"]
Sep 11 14:15:59.605806 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.605782    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m" podUID="25eac3ee-1027-496b-9a35-92cf639cb94a" containerName="payload-processing" containerID="cri-o://69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902" gracePeriod=30
Sep 11 14:15:59.624372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.623956    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"]
Sep 11 14:15:59.624372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.624200    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs" podUID="1fc8dc58-2e35-4327-8570-b6b8963bc6b2" containerName="payload-pre-processing" containerID="cri-o://eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076" gracePeriod=30
Sep 11 14:15:59.699153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.699117    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:59.876637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.876612    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:15:59.886399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.886378    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:15:59.888987 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.888969    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:15:59.953551 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.953519    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25eac3ee-1027-496b-9a35-92cf639cb94a-plugins-config-volume\") pod \"25eac3ee-1027-496b-9a35-92cf639cb94a\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") "
Sep 11 14:15:59.953705 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.953557    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-266w6\" (UniqueName: \"kubernetes.io/projected/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-kube-api-access-266w6\") pod \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") "
Sep 11 14:15:59.953705 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.953578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-876kx\" (UniqueName: \"kubernetes.io/projected/25eac3ee-1027-496b-9a35-92cf639cb94a-kube-api-access-876kx\") pod \"25eac3ee-1027-496b-9a35-92cf639cb94a\" (UID: \"25eac3ee-1027-496b-9a35-92cf639cb94a\") "
Sep 11 14:15:59.953705 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.953642    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-plugins-config-volume\") pod \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\" (UID: \"1fc8dc58-2e35-4327-8570-b6b8963bc6b2\") "
Sep 11 14:15:59.953993 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.953963    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25eac3ee-1027-496b-9a35-92cf639cb94a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25eac3ee-1027-496b-9a35-92cf639cb94a" (UID: "25eac3ee-1027-496b-9a35-92cf639cb94a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:15:59.954149 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.954047    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fc8dc58-2e35-4327-8570-b6b8963bc6b2" (UID: "1fc8dc58-2e35-4327-8570-b6b8963bc6b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:15:59.955740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.955713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25eac3ee-1027-496b-9a35-92cf639cb94a-kube-api-access-876kx" (OuterVolumeSpecName: "kube-api-access-876kx") pod "25eac3ee-1027-496b-9a35-92cf639cb94a" (UID: "25eac3ee-1027-496b-9a35-92cf639cb94a"). InnerVolumeSpecName "kube-api-access-876kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:15:59.955825 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.955764    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-kube-api-access-266w6" (OuterVolumeSpecName: "kube-api-access-266w6") pod "1fc8dc58-2e35-4327-8570-b6b8963bc6b2" (UID: "1fc8dc58-2e35-4327-8570-b6b8963bc6b2"). InnerVolumeSpecName "kube-api-access-266w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:15:59.983384 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:15:59.983350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:00.054805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.054766    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25eac3ee-1027-496b-9a35-92cf639cb94a-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:00.054805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.054797    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266w6\" (UniqueName: \"kubernetes.io/projected/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-kube-api-access-266w6\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:00.054805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.054806    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876kx\" (UniqueName: \"kubernetes.io/projected/25eac3ee-1027-496b-9a35-92cf639cb94a-kube-api-access-876kx\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:00.054805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.054814    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc8dc58-2e35-4327-8570-b6b8963bc6b2-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:00.098469 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.098379    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:00.141597 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.141554    2558 generic.go:356] "Generic (PLEG): container finished" podID="25eac3ee-1027-496b-9a35-92cf639cb94a" containerID="69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902" exitCode=0
Sep 11 14:16:00.141783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.141613    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"
Sep 11 14:16:00.141783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.141643    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m" event={"ID":"25eac3ee-1027-496b-9a35-92cf639cb94a","Type":"ContainerDied","Data":"69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902"}
Sep 11 14:16:00.141783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.141715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m" event={"ID":"25eac3ee-1027-496b-9a35-92cf639cb94a","Type":"ContainerDied","Data":"eab23ff69cf51e7506426e4f39f0a51ed046868f7a3e541bf546e721f1a836f7"}
Sep 11 14:16:00.141783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.141738    2558 scope.go:117] "RemoveContainer" containerID="69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902"
Sep 11 14:16:00.143452 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.143434    2558 generic.go:356] "Generic (PLEG): container finished" podID="1fc8dc58-2e35-4327-8570-b6b8963bc6b2" containerID="eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076" exitCode=0
Sep 11 14:16:00.143557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.143539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs" event={"ID":"1fc8dc58-2e35-4327-8570-b6b8963bc6b2","Type":"ContainerDied","Data":"eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076"}
Sep 11 14:16:00.143626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.143551    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"
Sep 11 14:16:00.143626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.143582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs" event={"ID":"1fc8dc58-2e35-4327-8570-b6b8963bc6b2","Type":"ContainerDied","Data":"44bba5f8638b282ff3aa24632710329d88666684dd87d4569be4f1e0a7f75c03"}
Sep 11 14:16:00.154400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.154368    2558 scope.go:117] "RemoveContainer" containerID="69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902"
Sep 11 14:16:00.154696 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:16:00.154662    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902\": container with ID starting with 69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902 not found: ID does not exist" containerID="69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902"
Sep 11 14:16:00.154779 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.154698    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902"} err="failed to get container status \"69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902\": rpc error: code = NotFound desc = could not find container \"69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902\": container with ID starting with 69a43ac3854eb2c5aa0a8a905a5c00393bb77457a62d355d66b6a256004ba902 not found: ID does not exist"
Sep 11 14:16:00.154779 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.154722    2558 scope.go:117] "RemoveContainer" containerID="eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076"
Sep 11 14:16:00.162991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.162968    2558 scope.go:117] "RemoveContainer" containerID="eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076"
Sep 11 14:16:00.163359 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:16:00.163336    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076\": container with ID starting with eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076 not found: ID does not exist" containerID="eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076"
Sep 11 14:16:00.163432 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.163368    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076"} err="failed to get container status \"eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076\": rpc error: code = NotFound desc = could not find container \"eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076\": container with ID starting with eead9bc2a96bd0427d64d0b3cf97685a6913d46ad288a75b1322effd69ce8076 not found: ID does not exist"
Sep 11 14:16:00.186773 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.186738    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"]
Sep 11 14:16:00.200959 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.200934    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d609a19a-88c5fb969-brz7m"]
Sep 11 14:16:00.233120 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.233093    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"]
Sep 11 14:16:00.237216 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.237193    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d609a19a-c89d86558-vkqfs"]
Sep 11 14:16:00.242753 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.242729    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:00.428613 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.428526    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:00.503374 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.503347    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc8dc58-2e35-4327-8570-b6b8963bc6b2" path="/var/lib/kubelet/pods/1fc8dc58-2e35-4327-8570-b6b8963bc6b2/volumes"
Sep 11 14:16:00.503757 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:00.503740    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25eac3ee-1027-496b-9a35-92cf639cb94a" path="/var/lib/kubelet/pods/25eac3ee-1027-496b-9a35-92cf639cb94a/volumes"
Sep 11 14:16:01.648227 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:01.648192    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:01.786136 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:01.786095    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:01.916103 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:01.915988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:02.725277 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:02.725251    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:03.267367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:03.267337    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:04.204016 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.203986    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"]
Sep 11 14:16:04.204422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.204173    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" podUID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" containerName="maas-api" containerID="cri-o://43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4" gracePeriod=30
Sep 11 14:16:04.275422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.275391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:04.455956 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.455894    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:16:04.493930 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.493891    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-metrics-certs\") pod \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") "
Sep 11 14:16:04.494097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.493961    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdrcq\" (UniqueName: \"kubernetes.io/projected/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-kube-api-access-hdrcq\") pod \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") "
Sep 11 14:16:04.494097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.493982    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-maas-api-tls\") pod \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") "
Sep 11 14:16:04.494097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.494007    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-openshift-service-ca\") pod \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\" (UID: \"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0\") "
Sep 11 14:16:04.494430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.494403    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" (UID: "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:16:04.496076 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.496051    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" (UID: "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:16:04.496525 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.496509    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" (UID: "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:16:04.496602 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.496529    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-kube-api-access-hdrcq" (OuterVolumeSpecName: "kube-api-access-hdrcq") pod "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" (UID: "ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0"). InnerVolumeSpecName "kube-api-access-hdrcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:16:04.595401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.595354    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdrcq\" (UniqueName: \"kubernetes.io/projected/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-kube-api-access-hdrcq\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:04.595401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.595397    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:04.595401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.595406    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:04.595401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.595418    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:04.605582 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.605554    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:04.790522 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.790435    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:04.899601 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:04.899565    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:05.170963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.170863    2558 generic.go:356] "Generic (PLEG): container finished" podID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" containerID="43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4" exitCode=0
Sep 11 14:16:05.170963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.170906    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"
Sep 11 14:16:05.170963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.170939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" event={"ID":"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0","Type":"ContainerDied","Data":"43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4"}
Sep 11 14:16:05.171233 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.170986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" event={"ID":"ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0","Type":"ContainerDied","Data":"3b3ade7556cb800c10e27e70eac7240d895a3bff25ce7a6643acf60adb763828"}
Sep 11 14:16:05.171233 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.171008    2558 scope.go:117] "RemoveContainer" containerID="43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4"
Sep 11 14:16:05.178975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.178960    2558 scope.go:117] "RemoveContainer" containerID="43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4"
Sep 11 14:16:05.179240 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:16:05.179221    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4\": container with ID starting with 43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4 not found: ID does not exist" containerID="43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4"
Sep 11 14:16:05.179332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.179246    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4"} err="failed to get container status \"43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4\": rpc error: code = NotFound desc = could not find container \"43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4\": container with ID starting with 43332575ab764bdfb12332132d175569aa869e1eda6ff4a9c4bf6744fa5ea9c4 not found: ID does not exist"
Sep 11 14:16:05.199178 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.199149    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"]
Sep 11 14:16:05.205629 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.205607    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv"]
Sep 11 14:16:05.350748 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.350693    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:05.551718 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:05.551690    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:06.002304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:06.002272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:16:06.503348 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:06.503318    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" path="/var/lib/kubelet/pods/ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0/volumes"
Sep 11 14:16:06.547175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:06.547148    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:07.450507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:07.450466    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2a21d9-6fbb7f6b6b-jxhhv" podUID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.31:8443/health\": context deadline exceeded"
Sep 11 14:16:08.080731 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.080674    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:08.489840 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.489813    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"]
Sep 11 14:16:08.490172 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490158    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" containerName="maas-api"
Sep 11 14:16:08.490228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490173    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" containerName="maas-api"
Sep 11 14:16:08.490228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490185    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25eac3ee-1027-496b-9a35-92cf639cb94a" containerName="payload-processing"
Sep 11 14:16:08.490228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490191    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="25eac3ee-1027-496b-9a35-92cf639cb94a" containerName="payload-processing"
Sep 11 14:16:08.490228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490202    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc8dc58-2e35-4327-8570-b6b8963bc6b2" containerName="payload-pre-processing"
Sep 11 14:16:08.490228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490208    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc8dc58-2e35-4327-8570-b6b8963bc6b2" containerName="payload-pre-processing"
Sep 11 14:16:08.490396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490272    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="25eac3ee-1027-496b-9a35-92cf639cb94a" containerName="payload-processing"
Sep 11 14:16:08.490396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490280    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc8dc58-2e35-4327-8570-b6b8963bc6b2" containerName="payload-pre-processing"
Sep 11 14:16:08.490396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.490290    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad4b52fc-1361-4d55-b5c8-a05ce9fbbfb0" containerName="maas-api"
Sep 11 14:16:08.494619 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.494601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.513009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.512990    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-be0c80\"/\"kube-root-ca.crt\""
Sep 11 14:16:08.513879 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.513849    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-be0c80\"/\"openshift-service-ca.crt\""
Sep 11 14:16:08.513935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.513889    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-be0c80\"/\"test-model-be0c80-kserve-self-signed-certs\""
Sep 11 14:16:08.513935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.513912    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-be0c80\"/\"default-dockercfg-99xrw\""
Sep 11 14:16:08.534157 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.534135    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"]
Sep 11 14:16:08.582893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.582867    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:08.633937 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.633912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-model-cache\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.634077 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.633947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-tmp-dir\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.634077 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.633967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182ef6c4-f600-4b55-8091-9401216f1f3c-tls-certs\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.634155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.634076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-dshm\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.634155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.634116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-home\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.634155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.634147    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6ml\" (UniqueName: \"kubernetes.io/projected/182ef6c4-f600-4b55-8091-9401216f1f3c-kube-api-access-jl6ml\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735395 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-tmp-dir\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182ef6c4-f600-4b55-8091-9401216f1f3c-tls-certs\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-dshm\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-home\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735620 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6ml\" (UniqueName: \"kubernetes.io/projected/182ef6c4-f600-4b55-8091-9401216f1f3c-kube-api-access-jl6ml\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735620 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-model-cache\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735929 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735905    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-home\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.735929 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.735919    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-model-cache\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.736117 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.736008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-tmp-dir\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.737547 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.737528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-dshm\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.737638 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.737624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182ef6c4-f600-4b55-8091-9401216f1f3c-tls-certs\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.747236 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.747186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6ml\" (UniqueName: \"kubernetes.io/projected/182ef6c4-f600-4b55-8091-9401216f1f3c-kube-api-access-jl6ml\") pod \"test-model-be0c80-kserve-75678d6db5-n5qxt\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") " pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:08.803315 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:08.803288    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:09.087268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:09.087191    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"]
Sep 11 14:16:09.089888 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:16:09.089864    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182ef6c4_f600_4b55_8091_9401216f1f3c.slice/crio-455b5f92cbcc2d1618de0ef5ccd1f9491f01e296eee7d4791e411f1818364780 WatchSource:0}: Error finding container 455b5f92cbcc2d1618de0ef5ccd1f9491f01e296eee7d4791e411f1818364780: Status 404 returned error can't find the container with id 455b5f92cbcc2d1618de0ef5ccd1f9491f01e296eee7d4791e411f1818364780
Sep 11 14:16:09.192361 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:09.192325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt" event={"ID":"182ef6c4-f600-4b55-8091-9401216f1f3c","Type":"ContainerStarted","Data":"a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9"}
Sep 11 14:16:09.192527 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:09.192371    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt" event={"ID":"182ef6c4-f600-4b55-8091-9401216f1f3c","Type":"ContainerStarted","Data":"455b5f92cbcc2d1618de0ef5ccd1f9491f01e296eee7d4791e411f1818364780"}
Sep 11 14:16:09.240843 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:09.240792    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt" podStartSLOduration=1.240777029 podStartE2EDuration="1.240777029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:09.240240606 +0000 UTC m=+1103.203855579" watchObservedRunningTime="2026-09-11 14:16:09.240777029 +0000 UTC m=+1103.204392000"
Sep 11 14:16:10.377301 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:10.377270    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:10.661446 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:10.661363    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"]
Sep 11 14:16:10.675586 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:10.675554    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:10.900868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:10.900836    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:11.378878 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:11.378841    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:18.803661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:18.803631    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:18.804089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:18.803692    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:18.815549 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:18.815525    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:19.256471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:19.256443    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:16:22.516935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:22.516908    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:16:25.679756 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:25.679695    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv" podUID="1234bb00-6f5e-4022-bee9-aba3e05a624c" containerName="main" containerID="cri-o://1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3" gracePeriod=45
Sep 11 14:16:25.926653 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:25.926630    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:16:26.086816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.086782    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-model-cache\") pod \"1234bb00-6f5e-4022-bee9-aba3e05a624c\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") "
Sep 11 14:16:26.087009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.086822    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-home\") pod \"1234bb00-6f5e-4022-bee9-aba3e05a624c\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") "
Sep 11 14:16:26.087009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.086864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btv54\" (UniqueName: \"kubernetes.io/projected/1234bb00-6f5e-4022-bee9-aba3e05a624c-kube-api-access-btv54\") pod \"1234bb00-6f5e-4022-bee9-aba3e05a624c\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") "
Sep 11 14:16:26.087009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.086933    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1234bb00-6f5e-4022-bee9-aba3e05a624c-tls-certs\") pod \"1234bb00-6f5e-4022-bee9-aba3e05a624c\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") "
Sep 11 14:16:26.087009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.086953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-dshm\") pod \"1234bb00-6f5e-4022-bee9-aba3e05a624c\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") "
Sep 11 14:16:26.087009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.086977    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-tmp-dir\") pod \"1234bb00-6f5e-4022-bee9-aba3e05a624c\" (UID: \"1234bb00-6f5e-4022-bee9-aba3e05a624c\") "
Sep 11 14:16:26.087297 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.087099    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1234bb00-6f5e-4022-bee9-aba3e05a624c" (UID: "1234bb00-6f5e-4022-bee9-aba3e05a624c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:26.087297 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.087122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-home" (OuterVolumeSpecName: "home") pod "1234bb00-6f5e-4022-bee9-aba3e05a624c" (UID: "1234bb00-6f5e-4022-bee9-aba3e05a624c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:26.087297 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.087290    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:26.087417 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.087307    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:26.087417 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.087304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1234bb00-6f5e-4022-bee9-aba3e05a624c" (UID: "1234bb00-6f5e-4022-bee9-aba3e05a624c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:26.089244 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.089224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-dshm" (OuterVolumeSpecName: "dshm") pod "1234bb00-6f5e-4022-bee9-aba3e05a624c" (UID: "1234bb00-6f5e-4022-bee9-aba3e05a624c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:26.089307 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.089244    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1234bb00-6f5e-4022-bee9-aba3e05a624c-kube-api-access-btv54" (OuterVolumeSpecName: "kube-api-access-btv54") pod "1234bb00-6f5e-4022-bee9-aba3e05a624c" (UID: "1234bb00-6f5e-4022-bee9-aba3e05a624c"). InnerVolumeSpecName "kube-api-access-btv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:16:26.089307 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.089276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1234bb00-6f5e-4022-bee9-aba3e05a624c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1234bb00-6f5e-4022-bee9-aba3e05a624c" (UID: "1234bb00-6f5e-4022-bee9-aba3e05a624c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:16:26.188749 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.188716    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv54\" (UniqueName: \"kubernetes.io/projected/1234bb00-6f5e-4022-bee9-aba3e05a624c-kube-api-access-btv54\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:26.188749 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.188740    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1234bb00-6f5e-4022-bee9-aba3e05a624c-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:26.188749 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.188750    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:26.188955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.188759    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1234bb00-6f5e-4022-bee9-aba3e05a624c-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:16:26.277550 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.277515    2558 generic.go:356] "Generic (PLEG): container finished" podID="1234bb00-6f5e-4022-bee9-aba3e05a624c" containerID="1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3" exitCode=0
Sep 11 14:16:26.277720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.277559    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"
Sep 11 14:16:26.277720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.277601    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv" event={"ID":"1234bb00-6f5e-4022-bee9-aba3e05a624c","Type":"ContainerDied","Data":"1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3"}
Sep 11 14:16:26.277720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.277647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv" event={"ID":"1234bb00-6f5e-4022-bee9-aba3e05a624c","Type":"ContainerDied","Data":"ef31f939dcca1a5a7b1f57c325e8c435e40188097ab57f172806dc6363c64957"}
Sep 11 14:16:26.277720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.277665    2558 scope.go:117] "RemoveContainer" containerID="1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3"
Sep 11 14:16:26.286221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.286206    2558 scope.go:117] "RemoveContainer" containerID="1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3"
Sep 11 14:16:26.286447 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:16:26.286432    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3\": container with ID starting with 1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3 not found: ID does not exist" containerID="1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3"
Sep 11 14:16:26.286494 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.286453    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3"} err="failed to get container status \"1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3\": rpc error: code = NotFound desc = could not find container \"1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3\": container with ID starting with 1f91d9d1b1a8a1b313e4a430578af6b5dfc4e045f2f866f871e7465d8ecd2dc3 not found: ID does not exist"
Sep 11 14:16:26.304800 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.304649    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"]
Sep 11 14:16:26.306286 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.306264    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2a21d9/autores-model-2a21d9-kserve-8bb7679f9-42vlv"]
Sep 11 14:16:26.503252 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:26.503217    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1234bb00-6f5e-4022-bee9-aba3e05a624c" path="/var/lib/kubelet/pods/1234bb00-6f5e-4022-bee9-aba3e05a624c/volumes"
Sep 11 14:16:31.468911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.468879    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"]
Sep 11 14:16:31.469304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.469281    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1234bb00-6f5e-4022-bee9-aba3e05a624c" containerName="main"
Sep 11 14:16:31.469304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.469293    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1234bb00-6f5e-4022-bee9-aba3e05a624c" containerName="main"
Sep 11 14:16:31.469381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.469351    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1234bb00-6f5e-4022-bee9-aba3e05a624c" containerName="main"
Sep 11 14:16:31.473904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.473885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.476507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.476492    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-c906f9bb-kserve-self-signed-certs\""
Sep 11 14:16:31.484755 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.484735    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"]
Sep 11 14:16:31.508910 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.508890    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:31.635223 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.635193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tls-certs\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.635325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.635248    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmv6f\" (UniqueName: \"kubernetes.io/projected/2b2c15d7-a681-4b25-89a4-8883c5bb048e-kube-api-access-jmv6f\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.635325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.635294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-dshm\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.635325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.635310    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tmp-dir\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.635437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.635379    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-home\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.635437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.635411    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-model-cache\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.736661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.736602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-home\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.736661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.736633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-model-cache\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.736816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.736659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tls-certs\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.736816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.736721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmv6f\" (UniqueName: \"kubernetes.io/projected/2b2c15d7-a681-4b25-89a4-8883c5bb048e-kube-api-access-jmv6f\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.736921 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.736860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-dshm\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.736921 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.736898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tmp-dir\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.737052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.737007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-model-cache\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.737052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.737019    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-home\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.737248 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.737225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tmp-dir\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.738921 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.738899    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-dshm\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.739153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.739137    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tls-certs\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:31.831742 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:31.831720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmv6f\" (UniqueName: \"kubernetes.io/projected/2b2c15d7-a681-4b25-89a4-8883c5bb048e-kube-api-access-jmv6f\") pod \"e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") " pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:32.083117 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:32.083056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:36.165206 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:16:36.165170    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b2c15d7_a681_4b25_89a4_8883c5bb048e.slice/crio-02a0f6f39c14d356142b073ce07e34cbb9c2955c93296791e664a7ba67939fa1 WatchSource:0}: Error finding container 02a0f6f39c14d356142b073ce07e34cbb9c2955c93296791e664a7ba67939fa1: Status 404 returned error can't find the container with id 02a0f6f39c14d356142b073ce07e34cbb9c2955c93296791e664a7ba67939fa1
Sep 11 14:16:36.167933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:36.167903    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"]
Sep 11 14:16:36.324173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:36.324136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9" event={"ID":"2b2c15d7-a681-4b25-89a4-8883c5bb048e","Type":"ContainerStarted","Data":"b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a"}
Sep 11 14:16:36.324320 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:36.324181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9" event={"ID":"2b2c15d7-a681-4b25-89a4-8883c5bb048e","Type":"ContainerStarted","Data":"02a0f6f39c14d356142b073ce07e34cbb9c2955c93296791e664a7ba67939fa1"}
Sep 11 14:16:42.083736 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:42.083696    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:42.084243 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:42.083752    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:42.096262 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:42.096237    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:42.364783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:42.364706    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:16:48.725918 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:48.725850    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9" podStartSLOduration=17.725833282 podStartE2EDuration="17.725833282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:43.505773534 +0000 UTC m=+1137.469388506" watchObservedRunningTime="2026-09-11 14:16:48.725833282 +0000 UTC m=+1142.689448255"
Sep 11 14:16:52.090617 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:52.090587    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:53.470400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:53.470371    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:53.638968 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:53.638910    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:56.209805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:56.209772    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:16:59.636850 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:59.636816    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:59.850193 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:59.850164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:16:59.971405 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:16:59.971374    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:01.234112 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:01.234075    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:01.436593 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:01.436563    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:01.781899 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:01.781860    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:02.094987 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:02.094895    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:02.477997 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:02.477966    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:03.080966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:03.080938    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:05.155330 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:05.155294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:05.744320 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:05.744292    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:05.914674 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:05.914643    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:06.684738 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:06.684707    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:08.291861 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:08.291830    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:09.403304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:09.403229    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:09.546975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:09.546945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:09.681552 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:09.681479    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:10.134927 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:10.134893    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:10.269367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:10.269333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:15.787746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:15.787708    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:17:15.791743 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:15.791717    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:19.537836 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:19.537802    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"]
Sep 11 14:17:19.582255 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:19.582224    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:19.928193 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:19.928117    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"]
Sep 11 14:17:20.014525 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:20.014488    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:22.611364 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:22.611297    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:22.765954 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:22.765917    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:22.898043 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:22.897956    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:32.367965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:32.367929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:33.382532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:33.382493    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:33.551187 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:33.551149    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:34.240988 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.240953    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:34.555933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.555851    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff" podUID="2f4a9ef1-773e-4202-99e3-0f4a10f7933d" containerName="main" containerID="cri-o://0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166" gracePeriod=45
Sep 11 14:17:34.576574 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.576548    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:34.794971 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.794948    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:17:34.943091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943068    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-dshm\") pod \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") "
Sep 11 14:17:34.943225 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943105    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tmp-dir\") pod \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") "
Sep 11 14:17:34.943225 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943137    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-home\") pod \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") "
Sep 11 14:17:34.943225 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943193    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5x4w\" (UniqueName: \"kubernetes.io/projected/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-kube-api-access-d5x4w\") pod \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") "
Sep 11 14:17:34.943385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943243    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tls-certs\") pod \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") "
Sep 11 14:17:34.943385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943276    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-model-cache\") pod \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\" (UID: \"2f4a9ef1-773e-4202-99e3-0f4a10f7933d\") "
Sep 11 14:17:34.943385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943338    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f4a9ef1-773e-4202-99e3-0f4a10f7933d" (UID: "2f4a9ef1-773e-4202-99e3-0f4a10f7933d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:34.943385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943366    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-home" (OuterVolumeSpecName: "home") pod "2f4a9ef1-773e-4202-99e3-0f4a10f7933d" (UID: "2f4a9ef1-773e-4202-99e3-0f4a10f7933d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:34.943579 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943525    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f4a9ef1-773e-4202-99e3-0f4a10f7933d" (UID: "2f4a9ef1-773e-4202-99e3-0f4a10f7933d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:34.943641 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943593    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.943641 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943609    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.943641 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.943622    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.945349 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.945329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-dshm" (OuterVolumeSpecName: "dshm") pod "2f4a9ef1-773e-4202-99e3-0f4a10f7933d" (UID: "2f4a9ef1-773e-4202-99e3-0f4a10f7933d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:34.945349 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.945340    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f4a9ef1-773e-4202-99e3-0f4a10f7933d" (UID: "2f4a9ef1-773e-4202-99e3-0f4a10f7933d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:17:34.945522 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.945380    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-kube-api-access-d5x4w" (OuterVolumeSpecName: "kube-api-access-d5x4w") pod "2f4a9ef1-773e-4202-99e3-0f4a10f7933d" (UID: "2f4a9ef1-773e-4202-99e3-0f4a10f7933d"). InnerVolumeSpecName "kube-api-access-d5x4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:17:34.945963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:34.945925    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9" podUID="2b2c15d7-a681-4b25-89a4-8883c5bb048e" containerName="main" containerID="cri-o://b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a" gracePeriod=45
Sep 11 14:17:35.043965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.043942    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5x4w\" (UniqueName: \"kubernetes.io/projected/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-kube-api-access-d5x4w\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.043965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.043964    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.044103 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.043973    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4a9ef1-773e-4202-99e3-0f4a10f7933d-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.171300 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.171274    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:17:35.245780 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.245723    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-dshm\") pod \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") "
Sep 11 14:17:35.245780 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.245762    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tls-certs\") pod \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") "
Sep 11 14:17:35.245950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.245809    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmv6f\" (UniqueName: \"kubernetes.io/projected/2b2c15d7-a681-4b25-89a4-8883c5bb048e-kube-api-access-jmv6f\") pod \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") "
Sep 11 14:17:35.245950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.245827    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tmp-dir\") pod \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") "
Sep 11 14:17:35.245950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.245844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-home\") pod \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") "
Sep 11 14:17:35.245950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.245861    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-model-cache\") pod \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\" (UID: \"2b2c15d7-a681-4b25-89a4-8883c5bb048e\") "
Sep 11 14:17:35.246234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.246157    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-home" (OuterVolumeSpecName: "home") pod "2b2c15d7-a681-4b25-89a4-8883c5bb048e" (UID: "2b2c15d7-a681-4b25-89a4-8883c5bb048e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:35.246234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.246170    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b2c15d7-a681-4b25-89a4-8883c5bb048e" (UID: "2b2c15d7-a681-4b25-89a4-8883c5bb048e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:35.246399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.246232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b2c15d7-a681-4b25-89a4-8883c5bb048e" (UID: "2b2c15d7-a681-4b25-89a4-8883c5bb048e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:35.246399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.246277    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.246399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.246296    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.247827 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.247804    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2c15d7-a681-4b25-89a4-8883c5bb048e-kube-api-access-jmv6f" (OuterVolumeSpecName: "kube-api-access-jmv6f") pod "2b2c15d7-a681-4b25-89a4-8883c5bb048e" (UID: "2b2c15d7-a681-4b25-89a4-8883c5bb048e"). InnerVolumeSpecName "kube-api-access-jmv6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:17:35.247916 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.247829    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-dshm" (OuterVolumeSpecName: "dshm") pod "2b2c15d7-a681-4b25-89a4-8883c5bb048e" (UID: "2b2c15d7-a681-4b25-89a4-8883c5bb048e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:35.247916 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.247833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b2c15d7-a681-4b25-89a4-8883c5bb048e" (UID: "2b2c15d7-a681-4b25-89a4-8883c5bb048e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:17:35.346966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.346941    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.346966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.346965    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2c15d7-a681-4b25-89a4-8883c5bb048e-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.347144 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.346977    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmv6f\" (UniqueName: \"kubernetes.io/projected/2b2c15d7-a681-4b25-89a4-8883c5bb048e-kube-api-access-jmv6f\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.347144 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.346986    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2c15d7-a681-4b25-89a4-8883c5bb048e-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.584485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.584455    2558 generic.go:356] "Generic (PLEG): container finished" podID="2f4a9ef1-773e-4202-99e3-0f4a10f7933d" containerID="0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166" exitCode=0
Sep 11 14:17:35.584829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.584510    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"
Sep 11 14:17:35.584829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.584539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff" event={"ID":"2f4a9ef1-773e-4202-99e3-0f4a10f7933d","Type":"ContainerDied","Data":"0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166"}
Sep 11 14:17:35.584829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.584585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff" event={"ID":"2f4a9ef1-773e-4202-99e3-0f4a10f7933d","Type":"ContainerDied","Data":"95d21fcf5afc445f48c16320a0d89fd3a9b959744c9868c281313e9653d1554e"}
Sep 11 14:17:35.584829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.584602    2558 scope.go:117] "RemoveContainer" containerID="0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166"
Sep 11 14:17:35.586120 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.586097    2558 generic.go:356] "Generic (PLEG): container finished" podID="2b2c15d7-a681-4b25-89a4-8883c5bb048e" containerID="b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a" exitCode=0
Sep 11 14:17:35.586217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.586157    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"
Sep 11 14:17:35.586217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.586181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9" event={"ID":"2b2c15d7-a681-4b25-89a4-8883c5bb048e","Type":"ContainerDied","Data":"b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a"}
Sep 11 14:17:35.586332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.586217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9" event={"ID":"2b2c15d7-a681-4b25-89a4-8883c5bb048e","Type":"ContainerDied","Data":"02a0f6f39c14d356142b073ce07e34cbb9c2955c93296791e664a7ba67939fa1"}
Sep 11 14:17:35.597248 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.597231    2558 scope.go:117] "RemoveContainer" containerID="0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166"
Sep 11 14:17:35.597573 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:17:35.597549    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166\": container with ID starting with 0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166 not found: ID does not exist" containerID="0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166"
Sep 11 14:17:35.597640 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.597585    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166"} err="failed to get container status \"0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166\": rpc error: code = NotFound desc = could not find container \"0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166\": container with ID starting with 0b0edbc0adfbbb897a7330fc05b0d1e139cac1c48871a6fd7c4bbe144f4b4166 not found: ID does not exist"
Sep 11 14:17:35.597640 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.597607    2558 scope.go:117] "RemoveContainer" containerID="b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a"
Sep 11 14:17:35.605436 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.605418    2558 scope.go:117] "RemoveContainer" containerID="b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a"
Sep 11 14:17:35.605681 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:17:35.605663    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a\": container with ID starting with b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a not found: ID does not exist" containerID="b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a"
Sep 11 14:17:35.605734 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.605690    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a"} err="failed to get container status \"b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a\": rpc error: code = NotFound desc = could not find container \"b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a\": container with ID starting with b4079262f2e63c1480a9410efd0e07940834e1c4a0bb85df03b9d91aef55f01a not found: ID does not exist"
Sep 11 14:17:35.628084 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.628059    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"]
Sep 11 14:17:35.636230 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.636208    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-c906f9bb-kserve-5b789f5b7d-ksxm9"]
Sep 11 14:17:35.655148 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.655117    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"]
Sep 11 14:17:35.687057 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.687023    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c906f9bb-kserve-6554dd85b8-9dsff"]
Sep 11 14:17:35.864785 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:35.864711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:36.040967 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:36.040937    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:36.503398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:36.503369    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2c15d7-a681-4b25-89a4-8883c5bb048e" path="/var/lib/kubelet/pods/2b2c15d7-a681-4b25-89a4-8883c5bb048e/volumes"
Sep 11 14:17:36.503729 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:36.503717    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4a9ef1-773e-4202-99e3-0f4a10f7933d" path="/var/lib/kubelet/pods/2f4a9ef1-773e-4202-99e3-0f4a10f7933d/volumes"
Sep 11 14:17:37.888728 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:37.888701    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:17:41.358217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:41.358182    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:41.489645 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:41.489609    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:42.532049 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:42.531995    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:43.001480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:43.001449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:44.767522 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:44.767490    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:46.504552 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:46.504525    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:17:46.505788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:46.505768    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:17:50.002363 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002333    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"]
Sep 11 14:17:50.002720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002673    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4a9ef1-773e-4202-99e3-0f4a10f7933d" containerName="main"
Sep 11 14:17:50.002720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002685    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4a9ef1-773e-4202-99e3-0f4a10f7933d" containerName="main"
Sep 11 14:17:50.002720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002710    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2c15d7-a681-4b25-89a4-8883c5bb048e" containerName="main"
Sep 11 14:17:50.002720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002715    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2c15d7-a681-4b25-89a4-8883c5bb048e" containerName="main"
Sep 11 14:17:50.002850 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002762    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4a9ef1-773e-4202-99e3-0f4a10f7933d" containerName="main"
Sep 11 14:17:50.002850 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.002771    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2c15d7-a681-4b25-89a4-8883c5bb048e" containerName="main"
Sep 11 14:17:50.007181 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.007162    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.011240 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.011222    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6156e\"/\"default-dockercfg-8sdrc\""
Sep 11 14:17:50.011332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.011222    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6156e\"/\"test-model-c6156e-kserve-self-signed-certs\""
Sep 11 14:17:50.011420 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.011400    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6156e\"/\"openshift-service-ca.crt\""
Sep 11 14:17:50.011484 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.011413    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6156e\"/\"kube-root-ca.crt\""
Sep 11 14:17:50.030480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.030454    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"]
Sep 11 14:17:50.063670 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.063641    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:50.163798 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.163766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-home\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.163888 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.163829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-tmp-dir\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.163888 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.163868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-dshm\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.163888 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.163883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-model-cache\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.164008 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.163948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08fdc89-37d5-4167-9378-a1ab43c2327b-tls-certs\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.164008 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.163989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvp9h\" (UniqueName: \"kubernetes.io/projected/d08fdc89-37d5-4167-9378-a1ab43c2327b-kube-api-access-rvp9h\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.264925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.264867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-home\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.264925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.264917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-tmp-dir\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265078 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.264949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-dshm\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265078 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.264965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-model-cache\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265148 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.265083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08fdc89-37d5-4167-9378-a1ab43c2327b-tls-certs\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265148 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.265133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvp9h\" (UniqueName: \"kubernetes.io/projected/d08fdc89-37d5-4167-9378-a1ab43c2327b-kube-api-access-rvp9h\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265321 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.265303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-home\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.265348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-model-cache\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.265437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.265394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-tmp-dir\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.267192 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.267168    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-dshm\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.267423 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.267409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08fdc89-37d5-4167-9378-a1ab43c2327b-tls-certs\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.277496 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.277471    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvp9h\" (UniqueName: \"kubernetes.io/projected/d08fdc89-37d5-4167-9378-a1ab43c2327b-kube-api-access-rvp9h\") pod \"test-model-c6156e-kserve-797595797c-7v2km\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") " pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.315751 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.315730    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:17:50.596259 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.596233    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"]
Sep 11 14:17:50.598527 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:17:50.598504    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd08fdc89_37d5_4167_9378_a1ab43c2327b.slice/crio-8965ebc4eccab74419b94d2c3c6207be7f30fada5002b731dd67a2aeef652769 WatchSource:0}: Error finding container 8965ebc4eccab74419b94d2c3c6207be7f30fada5002b731dd67a2aeef652769: Status 404 returned error can't find the container with id 8965ebc4eccab74419b94d2c3c6207be7f30fada5002b731dd67a2aeef652769
Sep 11 14:17:50.653662 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.653634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km" event={"ID":"d08fdc89-37d5-4167-9378-a1ab43c2327b","Type":"ContainerStarted","Data":"1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764"}
Sep 11 14:17:50.653780 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.653673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km" event={"ID":"d08fdc89-37d5-4167-9378-a1ab43c2327b","Type":"ContainerStarted","Data":"8965ebc4eccab74419b94d2c3c6207be7f30fada5002b731dd67a2aeef652769"}
Sep 11 14:17:50.679839 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:50.679787    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km" podStartSLOduration=1.679773517 podStartE2EDuration="1.679773517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:50.677820603 +0000 UTC m=+1204.641435577" watchObservedRunningTime="2026-09-11 14:17:50.679773517 +0000 UTC m=+1204.643388489"
Sep 11 14:17:52.083725 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:52.083693    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:52.218061 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:52.218012    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:53.382509 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:53.382473    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:53.497743 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:53.497714    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:53.802304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:53.802268    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:17:54.249491 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:54.249458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:58.804187 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.804140    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"]
Sep 11 14:17:58.807282 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.807263    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.810828 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.810805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bef04b\"/\"explicit-model-bef04b-kserve-self-signed-certs\""
Sep 11 14:17:58.810945 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.810805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bef04b\"/\"default-dockercfg-hpws9\""
Sep 11 14:17:58.811039 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.811009    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bef04b\"/\"openshift-service-ca.crt\""
Sep 11 14:17:58.811447 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.811428    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bef04b\"/\"kube-root-ca.crt\""
Sep 11 14:17:58.824823 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.824800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-dshm\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.824961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.824837    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1277e658-5a66-4310-80b8-b7fd07bcc14d-tls-certs\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.824961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.824853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-tmp-dir\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.824961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.824885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2lz\" (UniqueName: \"kubernetes.io/projected/1277e658-5a66-4310-80b8-b7fd07bcc14d-kube-api-access-bb2lz\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.824961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.824919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-model-cache\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.824961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.824952    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-home\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.825658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.825637    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"]
Sep 11 14:17:58.898986 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.898960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:17:58.925299 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2lz\" (UniqueName: \"kubernetes.io/projected/1277e658-5a66-4310-80b8-b7fd07bcc14d-kube-api-access-bb2lz\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-model-cache\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-home\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-dshm\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1277e658-5a66-4310-80b8-b7fd07bcc14d-tls-certs\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925402    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-tmp-dir\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925702 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-model-cache\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-home\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.925792 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.925755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-tmp-dir\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.927554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.927535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-dshm\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.927807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.927790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1277e658-5a66-4310-80b8-b7fd07bcc14d-tls-certs\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:58.935254 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:58.935231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2lz\" (UniqueName: \"kubernetes.io/projected/1277e658-5a66-4310-80b8-b7fd07bcc14d-kube-api-access-bb2lz\") pod \"explicit-model-bef04b-kserve-dd4df9c55-lbl5z\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") " pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:59.118981 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:59.118904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:17:59.396544 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:59.396474    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"]
Sep 11 14:17:59.399620 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:17:59.399587    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1277e658_5a66_4310_80b8_b7fd07bcc14d.slice/crio-e7921130fcf436a3e148e895ff81e632809853dc41bba12eb72140c0fa40689c WatchSource:0}: Error finding container e7921130fcf436a3e148e895ff81e632809853dc41bba12eb72140c0fa40689c: Status 404 returned error can't find the container with id e7921130fcf436a3e148e895ff81e632809853dc41bba12eb72140c0fa40689c
Sep 11 14:17:59.693161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:59.693127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z" event={"ID":"1277e658-5a66-4310-80b8-b7fd07bcc14d","Type":"ContainerStarted","Data":"b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5"}
Sep 11 14:17:59.693325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:59.693171    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z" event={"ID":"1277e658-5a66-4310-80b8-b7fd07bcc14d","Type":"ContainerStarted","Data":"e7921130fcf436a3e148e895ff81e632809853dc41bba12eb72140c0fa40689c"}
Sep 11 14:17:59.736752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:17:59.736701    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z" podStartSLOduration=1.7366850299999999 podStartE2EDuration="1.73668503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:59.736217965 +0000 UTC m=+1213.699832937" watchObservedRunningTime="2026-09-11 14:17:59.73668503 +0000 UTC m=+1213.700300002"
Sep 11 14:18:00.315862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:00.315827    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:18:00.316301 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:00.315875    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:18:00.327663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:00.327640    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:18:00.709329 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:00.709304    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:18:01.705214 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:01.705182    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:01.880214 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:01.880178    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:02.034741 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:02.034647    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:02.425667 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:02.425578    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:02.756210 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:02.756173    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:05.472832 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:05.472792    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:06.176351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:06.176319    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:06.317408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:06.317372    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:06.670013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:06.669982    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:07.922901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:07.922841    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:08.957683 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:08.957651    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:09.119318 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.119268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:18:09.119488 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.119342    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:18:09.131374 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.131348    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:18:09.750581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.750543    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:18:09.892364 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.892335    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"]
Sep 11 14:18:09.895851 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.895829    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:09.899022 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.899003    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-20588104-serving-cert\""
Sep 11 14:18:09.913894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.913864    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"]
Sep 11 14:18:09.921107 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.921082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:09.921195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.921129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-metrics-certs\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:09.921242 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.921204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smj9b\" (UniqueName: \"kubernetes.io/projected/67047f35-cc05-446b-9c11-ecae09d78dae-kube-api-access-smj9b\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:09.921299 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.921279    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f35-cc05-446b-9c11-ecae09d78dae-openshift-service-ca\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:09.941233 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:09.941209    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:10.022333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.022250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f35-cc05-446b-9c11-ecae09d78dae-openshift-service-ca\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.022333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.022308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.022758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.022341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-metrics-certs\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.022758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.022371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smj9b\" (UniqueName: \"kubernetes.io/projected/67047f35-cc05-446b-9c11-ecae09d78dae-kube-api-access-smj9b\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.022758 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:18:10.022463    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-20588104-serving-cert: secret "maas-api-e2e-mt-20588104-serving-cert" not found
Sep 11 14:18:10.022758 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:18:10.022551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls podName:67047f35-cc05-446b-9c11-ecae09d78dae nodeName:}" failed. No retries permitted until 2026-09-11 14:18:10.522530052 +0000 UTC m=+1224.486145014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls") pod "maas-api-e2e-mt-20588104-76449ccbbc-gmspf" (UID: "67047f35-cc05-446b-9c11-ecae09d78dae") : secret "maas-api-e2e-mt-20588104-serving-cert" not found
Sep 11 14:18:10.022955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.022935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f35-cc05-446b-9c11-ecae09d78dae-openshift-service-ca\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.024738 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.024719    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-metrics-certs\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.037807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.037780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:10.039577 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.039559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smj9b\" (UniqueName: \"kubernetes.io/projected/67047f35-cc05-446b-9c11-ecae09d78dae-kube-api-access-smj9b\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.478368 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.478333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:10.526481 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.526428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.529266 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.529238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls\") pod \"maas-api-e2e-mt-20588104-76449ccbbc-gmspf\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.601502 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.601469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:10.714853 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.714806    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:10.805799 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.805711    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:10.807991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.807967    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"]
Sep 11 14:18:10.811514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.811494    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:10.815717 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.815695    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-20588104-dockercfg-67w9b\""
Sep 11 14:18:10.815862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.815844    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-20588104\""
Sep 11 14:18:10.837952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.837926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"]
Sep 11 14:18:10.929663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.929632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgdt\" (UniqueName: \"kubernetes.io/projected/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-kube-api-access-rsgdt\") pod \"payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:10.929808 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:10.929680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.030765 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.030731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgdt\" (UniqueName: \"kubernetes.io/projected/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-kube-api-access-rsgdt\") pod \"payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.031260 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.030804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.031582 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.031557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.044756 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.044733    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgdt\" (UniqueName: \"kubernetes.io/projected/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-kube-api-access-rsgdt\") pod \"payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.092676 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.092655    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"]
Sep 11 14:18:11.095143 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:18:11.095119    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67047f35_cc05_446b_9c11_ecae09d78dae.slice/crio-b86bb00d19e2b22d8c14d63832901a3573df26811cc219446b0d6ceeb374e451 WatchSource:0}: Error finding container b86bb00d19e2b22d8c14d63832901a3573df26811cc219446b0d6ceeb374e451: Status 404 returned error can't find the container with id b86bb00d19e2b22d8c14d63832901a3573df26811cc219446b0d6ceeb374e451
Sep 11 14:18:11.096510 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.096491    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:18:11.138040 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.138011    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.419019 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.418952    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"]
Sep 11 14:18:11.422052 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:18:11.421998    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa940f3f_31cf_4f6e_84fe_c60533b3fd90.slice/crio-c109a75042c24c3fe9c2d0166f7249aa601602cef9913dbf777bcc14b0303f65 WatchSource:0}: Error finding container c109a75042c24c3fe9c2d0166f7249aa601602cef9913dbf777bcc14b0303f65: Status 404 returned error can't find the container with id c109a75042c24c3fe9c2d0166f7249aa601602cef9913dbf777bcc14b0303f65
Sep 11 14:18:11.749845 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.749810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf" event={"ID":"67047f35-cc05-446b-9c11-ecae09d78dae","Type":"ContainerStarted","Data":"ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14"}
Sep 11 14:18:11.750057 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.749879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf" event={"ID":"67047f35-cc05-446b-9c11-ecae09d78dae","Type":"ContainerStarted","Data":"b86bb00d19e2b22d8c14d63832901a3573df26811cc219446b0d6ceeb374e451"}
Sep 11 14:18:11.750057 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.749947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:11.751437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.751417    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9" event={"ID":"fa940f3f-31cf-4f6e-84fe-c60533b3fd90","Type":"ContainerStarted","Data":"13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198"}
Sep 11 14:18:11.751518 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.751453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9" event={"ID":"fa940f3f-31cf-4f6e-84fe-c60533b3fd90","Type":"ContainerStarted","Data":"c109a75042c24c3fe9c2d0166f7249aa601602cef9913dbf777bcc14b0303f65"}
Sep 11 14:18:11.751518 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.751472    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:11.782655 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.782596    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf" podStartSLOduration=2.362866429 podStartE2EDuration="2.782574291s" totalImagesPullingTime="419.707862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:11.780512059 +0000 UTC m=+1225.744127037" watchObservedRunningTime="2026-09-11 14:18:11.782574291 +0000 UTC m=+1225.746189269"
Sep 11 14:18:11.799174 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.799142    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:18:11.812565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:11.812518    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9" podStartSLOduration=1.812503731 podStartE2EDuration="1.812503731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:11.810939367 +0000 UTC m=+1225.774554351" watchObservedRunningTime="2026-09-11 14:18:11.812503731 +0000 UTC m=+1225.776118703"
Sep 11 14:18:15.059928 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:15.059895    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"]
Sep 11 14:18:15.116485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:15.116454    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:15.397554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:15.397461    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:16.612952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:16.612918    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:17.762536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:17.762510    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:17.925361 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:17.925310    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:18.952047 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:18.951993    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:20.076241 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:20.076202    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:21.136053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:21.136001    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:22.759234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:22.759203    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:26.064131 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:26.064096    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:26.191899 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:26.191869    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:26.298514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:26.298474    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:26.454156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:26.454126    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:26.688679 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:26.688631    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:27.076095 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:27.076057    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:27.482635 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:27.482607    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:27.679847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:27.679812    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:28.530851 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:28.530818    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:29.821727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:29.821691    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:30.078206 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.078114    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z" podUID="1277e658-5a66-4310-80b8-b7fd07bcc14d" containerName="main" containerID="cri-o://b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5" gracePeriod=45
Sep 11 14:18:30.325818 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.325792    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:18:30.412294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412202    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-home\") pod \"1277e658-5a66-4310-80b8-b7fd07bcc14d\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") "
Sep 11 14:18:30.412294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-model-cache\") pod \"1277e658-5a66-4310-80b8-b7fd07bcc14d\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") "
Sep 11 14:18:30.412532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412330    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-dshm\") pod \"1277e658-5a66-4310-80b8-b7fd07bcc14d\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") "
Sep 11 14:18:30.412532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412349    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-tmp-dir\") pod \"1277e658-5a66-4310-80b8-b7fd07bcc14d\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") "
Sep 11 14:18:30.412532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1277e658-5a66-4310-80b8-b7fd07bcc14d-tls-certs\") pod \"1277e658-5a66-4310-80b8-b7fd07bcc14d\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") "
Sep 11 14:18:30.412532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412408    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb2lz\" (UniqueName: \"kubernetes.io/projected/1277e658-5a66-4310-80b8-b7fd07bcc14d-kube-api-access-bb2lz\") pod \"1277e658-5a66-4310-80b8-b7fd07bcc14d\" (UID: \"1277e658-5a66-4310-80b8-b7fd07bcc14d\") "
Sep 11 14:18:30.412532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412493    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-home" (OuterVolumeSpecName: "home") pod "1277e658-5a66-4310-80b8-b7fd07bcc14d" (UID: "1277e658-5a66-4310-80b8-b7fd07bcc14d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:30.412777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412539    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1277e658-5a66-4310-80b8-b7fd07bcc14d" (UID: "1277e658-5a66-4310-80b8-b7fd07bcc14d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:30.412777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1277e658-5a66-4310-80b8-b7fd07bcc14d" (UID: "1277e658-5a66-4310-80b8-b7fd07bcc14d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:30.412777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412745    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:30.412777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412766    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:30.412910 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.412782    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:30.414581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.414556    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1277e658-5a66-4310-80b8-b7fd07bcc14d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1277e658-5a66-4310-80b8-b7fd07bcc14d" (UID: "1277e658-5a66-4310-80b8-b7fd07bcc14d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:18:30.414581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.414572    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1277e658-5a66-4310-80b8-b7fd07bcc14d-kube-api-access-bb2lz" (OuterVolumeSpecName: "kube-api-access-bb2lz") pod "1277e658-5a66-4310-80b8-b7fd07bcc14d" (UID: "1277e658-5a66-4310-80b8-b7fd07bcc14d"). InnerVolumeSpecName "kube-api-access-bb2lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:30.414760 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.414596    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-dshm" (OuterVolumeSpecName: "dshm") pod "1277e658-5a66-4310-80b8-b7fd07bcc14d" (UID: "1277e658-5a66-4310-80b8-b7fd07bcc14d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:30.513426 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.513401    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1277e658-5a66-4310-80b8-b7fd07bcc14d-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:30.513536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.513428    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1277e658-5a66-4310-80b8-b7fd07bcc14d-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:30.513536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.513444    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2lz\" (UniqueName: \"kubernetes.io/projected/1277e658-5a66-4310-80b8-b7fd07bcc14d-kube-api-access-bb2lz\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:30.839469 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.839437    2558 generic.go:356] "Generic (PLEG): container finished" podID="1277e658-5a66-4310-80b8-b7fd07bcc14d" containerID="b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5" exitCode=0
Sep 11 14:18:30.839911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.839480    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"
Sep 11 14:18:30.839911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.839521    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z" event={"ID":"1277e658-5a66-4310-80b8-b7fd07bcc14d","Type":"ContainerDied","Data":"b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5"}
Sep 11 14:18:30.839911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.839568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z" event={"ID":"1277e658-5a66-4310-80b8-b7fd07bcc14d","Type":"ContainerDied","Data":"e7921130fcf436a3e148e895ff81e632809853dc41bba12eb72140c0fa40689c"}
Sep 11 14:18:30.839911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.839591    2558 scope.go:117] "RemoveContainer" containerID="b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5"
Sep 11 14:18:30.849466 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.849431    2558 scope.go:117] "RemoveContainer" containerID="b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5"
Sep 11 14:18:30.850118 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:18:30.850091    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5\": container with ID starting with b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5 not found: ID does not exist" containerID="b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5"
Sep 11 14:18:30.850233 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.850123    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5"} err="failed to get container status \"b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5\": rpc error: code = NotFound desc = could not find container \"b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5\": container with ID starting with b4bc5d945ac475e8004ac21a1cb8d6f23c9f01648d7315eb2fda99128962fbc5 not found: ID does not exist"
Sep 11 14:18:30.856526 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.856505    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"]
Sep 11 14:18:30.863370 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:30.863349    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-bef04b/explicit-model-bef04b-kserve-dd4df9c55-lbl5z"]
Sep 11 14:18:31.834400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:31.834365    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:31.934735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:31.934695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:32.183300 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:32.183204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:18:32.503865 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:32.503823    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1277e658-5a66-4310-80b8-b7fd07bcc14d" path="/var/lib/kubelet/pods/1277e658-5a66-4310-80b8-b7fd07bcc14d/volumes"
Sep 11 14:18:32.827429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:32.827353    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:33.865143 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:33.865110    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:34.843947 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:34.843911    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:35.195292 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:35.195261    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:35.879396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:35.879345    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:36.048874 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:36.048838    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:36.272385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:36.272354    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:36.373500 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:36.373469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:36.497016 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:36.496980    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:36.607115 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:36.607016    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:37.068388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:37.068354    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:37.197407 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:37.197373    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:37.613408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:37.613369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:37.881111 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:37.881018    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:38.038688 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:38.038657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:43.751741 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:43.751657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:44.399666 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.399632    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"]
Sep 11 14:18:44.400014 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.400001    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1277e658-5a66-4310-80b8-b7fd07bcc14d" containerName="main"
Sep 11 14:18:44.400088 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.400017    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1277e658-5a66-4310-80b8-b7fd07bcc14d" containerName="main"
Sep 11 14:18:44.400126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.400096    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1277e658-5a66-4310-80b8-b7fd07bcc14d" containerName="main"
Sep 11 14:18:44.404574 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.404555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.408167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.408146    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ea85ff\"/\"kube-root-ca.crt\""
Sep 11 14:18:44.409165 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.409150    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ea85ff\"/\"default-dockercfg-j9svq\""
Sep 11 14:18:44.409247 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.409161    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ea85ff\"/\"openshift-service-ca.crt\""
Sep 11 14:18:44.410429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.410402    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ea85ff\"/\"noten-model-ea85ff-kserve-self-signed-certs\""
Sep 11 14:18:44.412286 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.412259    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:44.427650 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.427630    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"]
Sep 11 14:18:44.534076 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.534049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmnq\" (UniqueName: \"kubernetes.io/projected/d2e70db2-61b1-4c31-8219-be5eccae8639-kube-api-access-ntmnq\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.534206 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.534085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-model-cache\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.534206 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.534110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-dshm\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.534281 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.534212    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-home\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.534315 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.534276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-tmp-dir\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.534315 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.534301    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e70db2-61b1-4c31-8219-be5eccae8639-tls-certs\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.634818 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.634791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-dshm\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.634935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.634843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-home\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.634935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.634883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-tmp-dir\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.634935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.634902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e70db2-61b1-4c31-8219-be5eccae8639-tls-certs\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.635106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.635053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmnq\" (UniqueName: \"kubernetes.io/projected/d2e70db2-61b1-4c31-8219-be5eccae8639-kube-api-access-ntmnq\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.635106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.635094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-model-cache\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.635539 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.635516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-tmp-dir\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.635673 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.635568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-model-cache\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.635673 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.635596    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-home\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.637104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.637086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-dshm\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.637410 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.637393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e70db2-61b1-4c31-8219-be5eccae8639-tls-certs\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.649364 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.649340    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmnq\" (UniqueName: \"kubernetes.io/projected/d2e70db2-61b1-4c31-8219-be5eccae8639-kube-api-access-ntmnq\") pod \"noten-model-ea85ff-kserve-544cf46bc5-qt8hw\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") " pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.714137 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.714108    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:44.989873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:44.989847    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"]
Sep 11 14:18:44.991990 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:18:44.991967    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2e70db2_61b1_4c31_8219_be5eccae8639.slice/crio-77ba48f1ae0d618dcc860a4f4e9b4542050130627f08b74e664edd6a73e32c30 WatchSource:0}: Error finding container 77ba48f1ae0d618dcc860a4f4e9b4542050130627f08b74e664edd6a73e32c30: Status 404 returned error can't find the container with id 77ba48f1ae0d618dcc860a4f4e9b4542050130627f08b74e664edd6a73e32c30
Sep 11 14:18:45.078911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:45.078865    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:45.915221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:45.915180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw" event={"ID":"d2e70db2-61b1-4c31-8219-be5eccae8639","Type":"ContainerStarted","Data":"3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb"}
Sep 11 14:18:45.915221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:45.915231    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw" event={"ID":"d2e70db2-61b1-4c31-8219-be5eccae8639","Type":"ContainerStarted","Data":"77ba48f1ae0d618dcc860a4f4e9b4542050130627f08b74e664edd6a73e32c30"}
Sep 11 14:18:45.956236 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:45.956183    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw" podStartSLOduration=1.956166055 podStartE2EDuration="1.956166055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:45.954617566 +0000 UTC m=+1259.918232540" watchObservedRunningTime="2026-09-11 14:18:45.956166055 +0000 UTC m=+1259.919781046"
Sep 11 14:18:46.061637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:46.061603    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:46.166751 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:46.166656    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:46.739160 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:46.739131    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:47.982432 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:47.982395    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:49.239095 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:49.239053    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:49.879228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:49.879191    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:18:50.103821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:50.103766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:50.807862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:50.807815    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:50.967308 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:50.967272    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:51.230537 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:51.230505    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:54.715041 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:54.714997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:54.715500 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:54.715075    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:54.727601 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:54.727572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:54.971132 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:54.971050    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:18:57.190894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.190860    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"]
Sep 11 14:18:57.191398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.191093    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf" podUID="67047f35-cc05-446b-9c11-ecae09d78dae" containerName="maas-api" containerID="cri-o://ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14" gracePeriod=30
Sep 11 14:18:57.249896 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.249847    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:57.298965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.298926    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"]
Sep 11 14:18:57.299297 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.299245    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9" podUID="fa940f3f-31cf-4f6e-84fe-c60533b3fd90" containerName="payload-pre-processing" containerID="cri-o://13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198" gracePeriod=30
Sep 11 14:18:57.361660 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.361629    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:57.464427 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.464403    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:57.483557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.483528    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:57.547117 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.547092    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls\") pod \"67047f35-cc05-446b-9c11-ecae09d78dae\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") "
Sep 11 14:18:57.547251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.547176    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-metrics-certs\") pod \"67047f35-cc05-446b-9c11-ecae09d78dae\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") "
Sep 11 14:18:57.547251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.547210    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f35-cc05-446b-9c11-ecae09d78dae-openshift-service-ca\") pod \"67047f35-cc05-446b-9c11-ecae09d78dae\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") "
Sep 11 14:18:57.547362 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.547268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smj9b\" (UniqueName: \"kubernetes.io/projected/67047f35-cc05-446b-9c11-ecae09d78dae-kube-api-access-smj9b\") pod \"67047f35-cc05-446b-9c11-ecae09d78dae\" (UID: \"67047f35-cc05-446b-9c11-ecae09d78dae\") "
Sep 11 14:18:57.547650 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.547628    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67047f35-cc05-446b-9c11-ecae09d78dae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67047f35-cc05-446b-9c11-ecae09d78dae" (UID: "67047f35-cc05-446b-9c11-ecae09d78dae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:18:57.549102 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.548530    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67047f35-cc05-446b-9c11-ecae09d78dae-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:57.549636 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.549610    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67047f35-cc05-446b-9c11-ecae09d78dae" (UID: "67047f35-cc05-446b-9c11-ecae09d78dae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:18:57.549738 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.549632    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67047f35-cc05-446b-9c11-ecae09d78dae" (UID: "67047f35-cc05-446b-9c11-ecae09d78dae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:18:57.550316 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.550295    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67047f35-cc05-446b-9c11-ecae09d78dae-kube-api-access-smj9b" (OuterVolumeSpecName: "kube-api-access-smj9b") pod "67047f35-cc05-446b-9c11-ecae09d78dae" (UID: "67047f35-cc05-446b-9c11-ecae09d78dae"). InnerVolumeSpecName "kube-api-access-smj9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:57.552382 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.552363    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:57.589388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.589350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:57.649495 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.649465    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsgdt\" (UniqueName: \"kubernetes.io/projected/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-kube-api-access-rsgdt\") pod \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") "
Sep 11 14:18:57.649682 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.649548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-plugins-config-volume\") pod \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\" (UID: \"fa940f3f-31cf-4f6e-84fe-c60533b3fd90\") "
Sep 11 14:18:57.649898 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.649867    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:57.649898 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.649899    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67047f35-cc05-446b-9c11-ecae09d78dae-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:57.649898 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.649902    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa940f3f-31cf-4f6e-84fe-c60533b3fd90" (UID: "fa940f3f-31cf-4f6e-84fe-c60533b3fd90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:18:57.650186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.649914    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smj9b\" (UniqueName: \"kubernetes.io/projected/67047f35-cc05-446b-9c11-ecae09d78dae-kube-api-access-smj9b\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:57.651682 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.651661    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-kube-api-access-rsgdt" (OuterVolumeSpecName: "kube-api-access-rsgdt") pod "fa940f3f-31cf-4f6e-84fe-c60533b3fd90" (UID: "fa940f3f-31cf-4f6e-84fe-c60533b3fd90"). InnerVolumeSpecName "kube-api-access-rsgdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:57.698334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.698260    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:57.750686 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.750654    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsgdt\" (UniqueName: \"kubernetes.io/projected/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-kube-api-access-rsgdt\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:57.750686 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.750682    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa940f3f-31cf-4f6e-84fe-c60533b3fd90-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:18:57.976354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.976262    2558 generic.go:356] "Generic (PLEG): container finished" podID="67047f35-cc05-446b-9c11-ecae09d78dae" containerID="ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14" exitCode=0
Sep 11 14:18:57.976354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.976311    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"
Sep 11 14:18:57.976557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.976347    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf" event={"ID":"67047f35-cc05-446b-9c11-ecae09d78dae","Type":"ContainerDied","Data":"ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14"}
Sep 11 14:18:57.976557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.976400    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf" event={"ID":"67047f35-cc05-446b-9c11-ecae09d78dae","Type":"ContainerDied","Data":"b86bb00d19e2b22d8c14d63832901a3573df26811cc219446b0d6ceeb374e451"}
Sep 11 14:18:57.976557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.976423    2558 scope.go:117] "RemoveContainer" containerID="ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14"
Sep 11 14:18:57.977697 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.977675    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa940f3f-31cf-4f6e-84fe-c60533b3fd90" containerID="13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198" exitCode=0
Sep 11 14:18:57.977770 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.977753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9" event={"ID":"fa940f3f-31cf-4f6e-84fe-c60533b3fd90","Type":"ContainerDied","Data":"13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198"}
Sep 11 14:18:57.977821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.977797    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"
Sep 11 14:18:57.977893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.977798    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9" event={"ID":"fa940f3f-31cf-4f6e-84fe-c60533b3fd90","Type":"ContainerDied","Data":"c109a75042c24c3fe9c2d0166f7249aa601602cef9913dbf777bcc14b0303f65"}
Sep 11 14:18:57.993764 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.993735    2558 scope.go:117] "RemoveContainer" containerID="ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14"
Sep 11 14:18:57.994187 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:18:57.994157    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14\": container with ID starting with ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14 not found: ID does not exist" containerID="ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14"
Sep 11 14:18:57.994276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.994199    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14"} err="failed to get container status \"ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14\": rpc error: code = NotFound desc = could not find container \"ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14\": container with ID starting with ed844a5fad20e6e397996011088615a5787b6490e7ee8a9b09e24507aefbcc14 not found: ID does not exist"
Sep 11 14:18:57.994276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:57.994223    2558 scope.go:117] "RemoveContainer" containerID="13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198"
Sep 11 14:18:58.002156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.002138    2558 scope.go:117] "RemoveContainer" containerID="13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198"
Sep 11 14:18:58.002448 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:18:58.002430    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198\": container with ID starting with 13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198 not found: ID does not exist" containerID="13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198"
Sep 11 14:18:58.002494 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.002459    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198"} err="failed to get container status \"13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198\": rpc error: code = NotFound desc = could not find container \"13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198\": container with ID starting with 13358841c970420a95db47fd330ccd2713399fd02b76f48a17c460ecdb390198 not found: ID does not exist"
Sep 11 14:18:58.010229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.010200    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"]
Sep 11 14:18:58.016925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.016898    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20588104-69999448c8-4phd9"]
Sep 11 14:18:58.034097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.034064    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"]
Sep 11 14:18:58.038841 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.038815    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20588104-76449ccbbc-gmspf"]
Sep 11 14:18:58.185415 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.185376    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:18:58.503309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.503276    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67047f35-cc05-446b-9c11-ecae09d78dae" path="/var/lib/kubelet/pods/67047f35-cc05-446b-9c11-ecae09d78dae/volumes"
Sep 11 14:18:58.503677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:58.503625    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa940f3f-31cf-4f6e-84fe-c60533b3fd90" path="/var/lib/kubelet/pods/fa940f3f-31cf-4f6e-84fe-c60533b3fd90/volumes"
Sep 11 14:18:59.370290 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:18:59.370258    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:00.424274 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:00.424232    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:02.285895 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:02.285838    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:19:03.338046 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:03.337968    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:05.137497 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:05.137458    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"]
Sep 11 14:19:05.152527 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:05.152497    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:05.281658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:05.281626    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:05.512316 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:05.512284    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:05.659829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:05.659798    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:06.643926 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:06.643897    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"]
Sep 11 14:19:06.670712 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:06.670685    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:07.078178 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:07.078139    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:07.660792 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:07.660762    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"]
Sep 11 14:19:07.694750 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:07.694720    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:08.183243 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:08.183204    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:09.198531 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:09.198484    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:10.408154 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:10.408124    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:10.794063 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:10.794011    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:11.426984 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:11.426951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:11.796126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:11.796089    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:11.900612 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:11.900582    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:12.109341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:12.109257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:12.476945 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:12.476913    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:12.603752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:12.603711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:12.732901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:12.732820    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:15.178083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:15.178048    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:19:18.667490 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:18.667450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:18.794495 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:18.794459    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:18.900876 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:18.900839    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:19.001229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:19.001195    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:19.101862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:19.101830    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:19.491164 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:19.491135    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:19.650866 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:19.650835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:20.156214 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.156179    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw" podUID="d2e70db2-61b1-4c31-8219-be5eccae8639" containerName="main" containerID="cri-o://3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb" gracePeriod=45
Sep 11 14:19:20.408225 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.408167    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:19:20.530072 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.530015    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:20.559079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e70db2-61b1-4c31-8219-be5eccae8639-tls-certs\") pod \"d2e70db2-61b1-4c31-8219-be5eccae8639\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") "
Sep 11 14:19:20.559213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559096    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-tmp-dir\") pod \"d2e70db2-61b1-4c31-8219-be5eccae8639\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") "
Sep 11 14:19:20.559213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559158    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-home\") pod \"d2e70db2-61b1-4c31-8219-be5eccae8639\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") "
Sep 11 14:19:20.559213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559195    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-model-cache\") pod \"d2e70db2-61b1-4c31-8219-be5eccae8639\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") "
Sep 11 14:19:20.559369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559231    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntmnq\" (UniqueName: \"kubernetes.io/projected/d2e70db2-61b1-4c31-8219-be5eccae8639-kube-api-access-ntmnq\") pod \"d2e70db2-61b1-4c31-8219-be5eccae8639\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") "
Sep 11 14:19:20.559369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559278    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-dshm\") pod \"d2e70db2-61b1-4c31-8219-be5eccae8639\" (UID: \"d2e70db2-61b1-4c31-8219-be5eccae8639\") "
Sep 11 14:19:20.559474 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559387    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2e70db2-61b1-4c31-8219-be5eccae8639" (UID: "d2e70db2-61b1-4c31-8219-be5eccae8639"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:20.559474 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559427    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-home" (OuterVolumeSpecName: "home") pod "d2e70db2-61b1-4c31-8219-be5eccae8639" (UID: "d2e70db2-61b1-4c31-8219-be5eccae8639"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:20.559584 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559519    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2e70db2-61b1-4c31-8219-be5eccae8639" (UID: "d2e70db2-61b1-4c31-8219-be5eccae8639"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:20.559670 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559653    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:20.559726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559675    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:20.559726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.559688    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:20.561435 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.561404    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-dshm" (OuterVolumeSpecName: "dshm") pod "d2e70db2-61b1-4c31-8219-be5eccae8639" (UID: "d2e70db2-61b1-4c31-8219-be5eccae8639"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:20.561536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.561430    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e70db2-61b1-4c31-8219-be5eccae8639-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2e70db2-61b1-4c31-8219-be5eccae8639" (UID: "d2e70db2-61b1-4c31-8219-be5eccae8639"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:20.561536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.561483    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e70db2-61b1-4c31-8219-be5eccae8639-kube-api-access-ntmnq" (OuterVolumeSpecName: "kube-api-access-ntmnq") pod "d2e70db2-61b1-4c31-8219-be5eccae8639" (UID: "d2e70db2-61b1-4c31-8219-be5eccae8639"). InnerVolumeSpecName "kube-api-access-ntmnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:20.661121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.661009    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e70db2-61b1-4c31-8219-be5eccae8639-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:20.661121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.661073    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntmnq\" (UniqueName: \"kubernetes.io/projected/d2e70db2-61b1-4c31-8219-be5eccae8639-kube-api-access-ntmnq\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:20.661121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.661084    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e70db2-61b1-4c31-8219-be5eccae8639-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:20.765213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.765168    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:20.904486 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:20.904446    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:21.020050 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.020004    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:21.094554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.094519    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2e70db2-61b1-4c31-8219-be5eccae8639" containerID="3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb" exitCode=0
Sep 11 14:19:21.094722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.094574    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"
Sep 11 14:19:21.094722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.094597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw" event={"ID":"d2e70db2-61b1-4c31-8219-be5eccae8639","Type":"ContainerDied","Data":"3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb"}
Sep 11 14:19:21.094722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.094641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw" event={"ID":"d2e70db2-61b1-4c31-8219-be5eccae8639","Type":"ContainerDied","Data":"77ba48f1ae0d618dcc860a4f4e9b4542050130627f08b74e664edd6a73e32c30"}
Sep 11 14:19:21.094722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.094660    2558 scope.go:117] "RemoveContainer" containerID="3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb"
Sep 11 14:19:21.104211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.104188    2558 scope.go:117] "RemoveContainer" containerID="3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb"
Sep 11 14:19:21.104541 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:19:21.104520    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb\": container with ID starting with 3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb not found: ID does not exist" containerID="3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb"
Sep 11 14:19:21.104600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.104551    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb"} err="failed to get container status \"3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb\": rpc error: code = NotFound desc = could not find container \"3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb\": container with ID starting with 3a8866eb138551226ddeef01055a7439a0c9653573d0bc0ee81aa90539b555fb not found: ID does not exist"
Sep 11 14:19:21.120338 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.120305    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:21.120641 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.120612    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"]
Sep 11 14:19:21.126990 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.126968    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ea85ff/noten-model-ea85ff-kserve-544cf46bc5-qt8hw"]
Sep 11 14:19:21.662394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.662350    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt" podUID="182ef6c4-f600-4b55-8091-9401216f1f3c" containerName="main" containerID="cri-o://a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9" gracePeriod=45
Sep 11 14:19:21.916677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:21.916616    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:19:22.075645 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075613    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-dshm\") pod \"182ef6c4-f600-4b55-8091-9401216f1f3c\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") "
Sep 11 14:19:22.075847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075669    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-model-cache\") pod \"182ef6c4-f600-4b55-8091-9401216f1f3c\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") "
Sep 11 14:19:22.075847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075690    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl6ml\" (UniqueName: \"kubernetes.io/projected/182ef6c4-f600-4b55-8091-9401216f1f3c-kube-api-access-jl6ml\") pod \"182ef6c4-f600-4b55-8091-9401216f1f3c\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") "
Sep 11 14:19:22.075847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075713    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-tmp-dir\") pod \"182ef6c4-f600-4b55-8091-9401216f1f3c\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") "
Sep 11 14:19:22.075847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182ef6c4-f600-4b55-8091-9401216f1f3c-tls-certs\") pod \"182ef6c4-f600-4b55-8091-9401216f1f3c\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") "
Sep 11 14:19:22.075847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-home\") pod \"182ef6c4-f600-4b55-8091-9401216f1f3c\" (UID: \"182ef6c4-f600-4b55-8091-9401216f1f3c\") "
Sep 11 14:19:22.076154 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.075944    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "182ef6c4-f600-4b55-8091-9401216f1f3c" (UID: "182ef6c4-f600-4b55-8091-9401216f1f3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:22.076154 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.076084    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "182ef6c4-f600-4b55-8091-9401216f1f3c" (UID: "182ef6c4-f600-4b55-8091-9401216f1f3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:22.076154 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.076145    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:22.076279 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.076161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-home" (OuterVolumeSpecName: "home") pod "182ef6c4-f600-4b55-8091-9401216f1f3c" (UID: "182ef6c4-f600-4b55-8091-9401216f1f3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:22.077983 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.077961    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182ef6c4-f600-4b55-8091-9401216f1f3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "182ef6c4-f600-4b55-8091-9401216f1f3c" (UID: "182ef6c4-f600-4b55-8091-9401216f1f3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:22.078091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.077966    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-dshm" (OuterVolumeSpecName: "dshm") pod "182ef6c4-f600-4b55-8091-9401216f1f3c" (UID: "182ef6c4-f600-4b55-8091-9401216f1f3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:22.078091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.078063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182ef6c4-f600-4b55-8091-9401216f1f3c-kube-api-access-jl6ml" (OuterVolumeSpecName: "kube-api-access-jl6ml") pod "182ef6c4-f600-4b55-8091-9401216f1f3c" (UID: "182ef6c4-f600-4b55-8091-9401216f1f3c"). InnerVolumeSpecName "kube-api-access-jl6ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:22.101938 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.101905    2558 generic.go:356] "Generic (PLEG): container finished" podID="182ef6c4-f600-4b55-8091-9401216f1f3c" containerID="a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9" exitCode=0
Sep 11 14:19:22.102118 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.101944    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"
Sep 11 14:19:22.102118 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.101996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt" event={"ID":"182ef6c4-f600-4b55-8091-9401216f1f3c","Type":"ContainerDied","Data":"a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9"}
Sep 11 14:19:22.102118 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.102065    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt" event={"ID":"182ef6c4-f600-4b55-8091-9401216f1f3c","Type":"ContainerDied","Data":"455b5f92cbcc2d1618de0ef5ccd1f9491f01e296eee7d4791e411f1818364780"}
Sep 11 14:19:22.102118 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.102088    2558 scope.go:117] "RemoveContainer" containerID="a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9"
Sep 11 14:19:22.110618 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.110601    2558 scope.go:117] "RemoveContainer" containerID="a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9"
Sep 11 14:19:22.110895 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:19:22.110876    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9\": container with ID starting with a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9 not found: ID does not exist" containerID="a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9"
Sep 11 14:19:22.110941 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.110902    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9"} err="failed to get container status \"a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9\": rpc error: code = NotFound desc = could not find container \"a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9\": container with ID starting with a88d152da4a2e96319d77b88fadc8fd242bdc0c2eb31d561bfb2750cf9dc1dc9 not found: ID does not exist"
Sep 11 14:19:22.157589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.157549    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"]
Sep 11 14:19:22.171103 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.171017    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-be0c80/test-model-be0c80-kserve-75678d6db5-n5qxt"]
Sep 11 14:19:22.176838 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.176817    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182ef6c4-f600-4b55-8091-9401216f1f3c-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:22.176904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.176840    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:22.176904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.176849    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:22.176904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.176858    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl6ml\" (UniqueName: \"kubernetes.io/projected/182ef6c4-f600-4b55-8091-9401216f1f3c-kube-api-access-jl6ml\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:22.176904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.176867    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182ef6c4-f600-4b55-8091-9401216f1f3c-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:22.504091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.504055    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182ef6c4-f600-4b55-8091-9401216f1f3c" path="/var/lib/kubelet/pods/182ef6c4-f600-4b55-8091-9401216f1f3c/volumes"
Sep 11 14:19:22.504453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.504439    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e70db2-61b1-4c31-8219-be5eccae8639" path="/var/lib/kubelet/pods/d2e70db2-61b1-4c31-8219-be5eccae8639/volumes"
Sep 11 14:19:22.589620 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.589587    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:19:22.679422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.679384    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km" podUID="d08fdc89-37d5-4167-9378-a1ab43c2327b" containerName="main" containerID="cri-o://1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764" gracePeriod=45
Sep 11 14:19:22.928513 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:22.928486    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:19:23.086392 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvp9h\" (UniqueName: \"kubernetes.io/projected/d08fdc89-37d5-4167-9378-a1ab43c2327b-kube-api-access-rvp9h\") pod \"d08fdc89-37d5-4167-9378-a1ab43c2327b\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") "
Sep 11 14:19:23.086392 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086360    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08fdc89-37d5-4167-9378-a1ab43c2327b-tls-certs\") pod \"d08fdc89-37d5-4167-9378-a1ab43c2327b\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") "
Sep 11 14:19:23.086643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086410    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-home\") pod \"d08fdc89-37d5-4167-9378-a1ab43c2327b\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") "
Sep 11 14:19:23.086643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086434    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-model-cache\") pod \"d08fdc89-37d5-4167-9378-a1ab43c2327b\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") "
Sep 11 14:19:23.086643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086462    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-tmp-dir\") pod \"d08fdc89-37d5-4167-9378-a1ab43c2327b\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") "
Sep 11 14:19:23.086643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086549    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-dshm\") pod \"d08fdc89-37d5-4167-9378-a1ab43c2327b\" (UID: \"d08fdc89-37d5-4167-9378-a1ab43c2327b\") "
Sep 11 14:19:23.086857 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086730    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-home" (OuterVolumeSpecName: "home") pod "d08fdc89-37d5-4167-9378-a1ab43c2327b" (UID: "d08fdc89-37d5-4167-9378-a1ab43c2327b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:23.086857 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086744    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d08fdc89-37d5-4167-9378-a1ab43c2327b" (UID: "d08fdc89-37d5-4167-9378-a1ab43c2327b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:23.086857 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086764    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d08fdc89-37d5-4167-9378-a1ab43c2327b" (UID: "d08fdc89-37d5-4167-9378-a1ab43c2327b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:23.086979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086865    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:23.086979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086883    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:23.086979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.086897    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:23.088716 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.088694    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-dshm" (OuterVolumeSpecName: "dshm") pod "d08fdc89-37d5-4167-9378-a1ab43c2327b" (UID: "d08fdc89-37d5-4167-9378-a1ab43c2327b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:23.088716 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.088707    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08fdc89-37d5-4167-9378-a1ab43c2327b-kube-api-access-rvp9h" (OuterVolumeSpecName: "kube-api-access-rvp9h") pod "d08fdc89-37d5-4167-9378-a1ab43c2327b" (UID: "d08fdc89-37d5-4167-9378-a1ab43c2327b"). InnerVolumeSpecName "kube-api-access-rvp9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:23.088822 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.088713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08fdc89-37d5-4167-9378-a1ab43c2327b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d08fdc89-37d5-4167-9378-a1ab43c2327b" (UID: "d08fdc89-37d5-4167-9378-a1ab43c2327b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:23.108738 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.108700    2558 generic.go:356] "Generic (PLEG): container finished" podID="d08fdc89-37d5-4167-9378-a1ab43c2327b" containerID="1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764" exitCode=0
Sep 11 14:19:23.108875 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.108746    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km" event={"ID":"d08fdc89-37d5-4167-9378-a1ab43c2327b","Type":"ContainerDied","Data":"1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764"}
Sep 11 14:19:23.108875 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.108776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km" event={"ID":"d08fdc89-37d5-4167-9378-a1ab43c2327b","Type":"ContainerDied","Data":"8965ebc4eccab74419b94d2c3c6207be7f30fada5002b731dd67a2aeef652769"}
Sep 11 14:19:23.108875 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.108792    2558 scope.go:117] "RemoveContainer" containerID="1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764"
Sep 11 14:19:23.108875 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.108750    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"
Sep 11 14:19:23.117511 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.117489    2558 scope.go:117] "RemoveContainer" containerID="1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764"
Sep 11 14:19:23.117811 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:19:23.117793    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764\": container with ID starting with 1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764 not found: ID does not exist" containerID="1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764"
Sep 11 14:19:23.117854 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.117818    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764"} err="failed to get container status \"1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764\": rpc error: code = NotFound desc = could not find container \"1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764\": container with ID starting with 1d3a442617cbd14113eff58ed6fcc055c209e5af637b882413ced8552346b764 not found: ID does not exist"
Sep 11 14:19:23.141449 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.141409    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"]
Sep 11 14:19:23.147678 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.147653    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c6156e/test-model-c6156e-kserve-797595797c-7v2km"]
Sep 11 14:19:23.188179 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.188144    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08fdc89-37d5-4167-9378-a1ab43c2327b-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:23.188179 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.188171    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvp9h\" (UniqueName: \"kubernetes.io/projected/d08fdc89-37d5-4167-9378-a1ab43c2327b-kube-api-access-rvp9h\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:23.188179 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:23.188180    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08fdc89-37d5-4167-9378-a1ab43c2327b-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:24.503771 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:24.503739    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08fdc89-37d5-4167-9378-a1ab43c2327b" path="/var/lib/kubelet/pods/d08fdc89-37d5-4167-9378-a1ab43c2327b/volumes"
Sep 11 14:19:25.888228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:25.888198    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:25.988674 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:25.988636    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:26.160161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:26.159974    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:27.681745 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:27.681708    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:28.713603 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:28.713561    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:29.289093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:29.289056    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:29.390095 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:29.390061    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:30.313600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:30.313557    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:30.646352 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:30.646257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:30.984822 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:30.984790    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:33.617123 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:33.617085    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:33.729183 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:33.729146    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:35.125476 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:35.125438    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:35.256669 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:35.256628    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:35.399760 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:35.399653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:35.790801 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:35.790766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:35.883910 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:35.883871    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:36.070800 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.070705    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"]
Sep 11 14:19:36.071172 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071153    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67047f35-cc05-446b-9c11-ecae09d78dae" containerName="maas-api"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071175    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="67047f35-cc05-446b-9c11-ecae09d78dae" containerName="maas-api"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071197    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08fdc89-37d5-4167-9378-a1ab43c2327b" containerName="main"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071205    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08fdc89-37d5-4167-9378-a1ab43c2327b" containerName="main"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071221    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182ef6c4-f600-4b55-8091-9401216f1f3c" containerName="main"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071229    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="182ef6c4-f600-4b55-8091-9401216f1f3c" containerName="main"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071257    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa940f3f-31cf-4f6e-84fe-c60533b3fd90" containerName="payload-pre-processing"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071266    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa940f3f-31cf-4f6e-84fe-c60533b3fd90" containerName="payload-pre-processing"
Sep 11 14:19:36.071276 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071275    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e70db2-61b1-4c31-8219-be5eccae8639" containerName="main"
Sep 11 14:19:36.071654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071284    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e70db2-61b1-4c31-8219-be5eccae8639" containerName="main"
Sep 11 14:19:36.071654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071365    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e70db2-61b1-4c31-8219-be5eccae8639" containerName="main"
Sep 11 14:19:36.071654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071377    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa940f3f-31cf-4f6e-84fe-c60533b3fd90" containerName="payload-pre-processing"
Sep 11 14:19:36.071654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071386    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08fdc89-37d5-4167-9378-a1ab43c2327b" containerName="main"
Sep 11 14:19:36.071654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071394    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="182ef6c4-f600-4b55-8091-9401216f1f3c" containerName="main"
Sep 11 14:19:36.071654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.071403    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="67047f35-cc05-446b-9c11-ecae09d78dae" containerName="maas-api"
Sep 11 14:19:36.075972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.075950    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.078520 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.078497    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fb993c\""
Sep 11 14:19:36.078621 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.078529    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-fb993c-dockercfg-w6smg\""
Sep 11 14:19:36.093242 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.093213    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"]
Sep 11 14:19:36.100624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.100597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtpf\" (UniqueName: \"kubernetes.io/projected/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-kube-api-access-phtpf\") pod \"payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.100764 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.100638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.202136 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.202093    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtpf\" (UniqueName: \"kubernetes.io/projected/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-kube-api-access-phtpf\") pod \"payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.202523 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.202152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.202878 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.202856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.247790 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.247756    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtpf\" (UniqueName: \"kubernetes.io/projected/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-kube-api-access-phtpf\") pod \"payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.386545 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.386448    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:36.669359 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.669281    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"]
Sep 11 14:19:36.672052 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:19:36.672001    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ecb7e6_9369_4606_97c9_f18f93aa9ba7.slice/crio-9e75104ad27da8262814fe972a6f1626230700b7ff0f41edd5ab87bb3916168e WatchSource:0}: Error finding container 9e75104ad27da8262814fe972a6f1626230700b7ff0f41edd5ab87bb3916168e: Status 404 returned error can't find the container with id 9e75104ad27da8262814fe972a6f1626230700b7ff0f41edd5ab87bb3916168e
Sep 11 14:19:36.755726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:36.755686    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:37.179845 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.179806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh" event={"ID":"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7","Type":"ContainerStarted","Data":"563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6"}
Sep 11 14:19:37.180059 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.179865    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh" event={"ID":"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7","Type":"ContainerStarted","Data":"9e75104ad27da8262814fe972a6f1626230700b7ff0f41edd5ab87bb3916168e"}
Sep 11 14:19:37.180059 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.179897    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:37.301602 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.301554    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh" podStartSLOduration=1.301537983 podStartE2EDuration="1.301537983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:37.273439585 +0000 UTC m=+1311.237054648" watchObservedRunningTime="2026-09-11 14:19:37.301537983 +0000 UTC m=+1311.265152956"
Sep 11 14:19:37.303388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.303366    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"]
Sep 11 14:19:37.303603 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.303581    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" podUID="6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" containerName="maas-api" containerID="cri-o://22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45" gracePeriod=30
Sep 11 14:19:37.368685 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.368652    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:37.500822 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.500783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"]
Sep 11 14:19:37.501010 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.500987    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk" podUID="9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" containerName="payload-pre-processing" containerID="cri-o://d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520" gracePeriod=30
Sep 11 14:19:37.593390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.593357    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:37.625659 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.625632    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:19:37.719547 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.719505    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5xj5\" (UniqueName: \"kubernetes.io/projected/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-kube-api-access-p5xj5\") pod \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") "
Sep 11 14:19:37.719717 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.719572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-metrics-certs\") pod \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") "
Sep 11 14:19:37.719717 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.719638    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls\") pod \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") "
Sep 11 14:19:37.719853 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.719719    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-openshift-service-ca\") pod \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\" (UID: \"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6\") "
Sep 11 14:19:37.720347 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.720317    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" (UID: "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:19:37.723912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.723888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" (UID: "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:37.723997 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.723941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" (UID: "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:37.723997 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.723957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-kube-api-access-p5xj5" (OuterVolumeSpecName: "kube-api-access-p5xj5") pod "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" (UID: "6dfd9a8e-ecd0-467c-a382-5c2ee44124b6"). InnerVolumeSpecName "kube-api-access-p5xj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:37.749703 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.749680    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:19:37.773398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.773366    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:37.820809 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.820769    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-plugins-config-volume\") pod \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") "
Sep 11 14:19:37.820992 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.820902    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfhbc\" (UniqueName: \"kubernetes.io/projected/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-kube-api-access-cfhbc\") pod \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\" (UID: \"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5\") "
Sep 11 14:19:37.821175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.821141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" (UID: "9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:19:37.821268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.821252    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5xj5\" (UniqueName: \"kubernetes.io/projected/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-kube-api-access-p5xj5\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:37.821317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.821269    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:37.821317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.821278    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:37.821317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.821288    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:37.821317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.821303    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:37.823011 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.822990    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-kube-api-access-cfhbc" (OuterVolumeSpecName: "kube-api-access-cfhbc") pod "9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" (UID: "9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5"). InnerVolumeSpecName "kube-api-access-cfhbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:37.922544 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.922505    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhbc\" (UniqueName: \"kubernetes.io/projected/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5-kube-api-access-cfhbc\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:19:37.945253 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:37.945215    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:38.048262 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.048160    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:38.186586 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.186554    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" containerID="d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520" exitCode=0
Sep 11 14:19:38.186788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.186609    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"
Sep 11 14:19:38.186788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.186632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk" event={"ID":"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5","Type":"ContainerDied","Data":"d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520"}
Sep 11 14:19:38.186788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.186682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk" event={"ID":"9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5","Type":"ContainerDied","Data":"caf21c4d2b85cc541485a37967908a2ad6b025f728040dca874e864be1882391"}
Sep 11 14:19:38.186788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.186704    2558 scope.go:117] "RemoveContainer" containerID="d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520"
Sep 11 14:19:38.188196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.188168    2558 generic.go:356] "Generic (PLEG): container finished" podID="6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" containerID="22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45" exitCode=0
Sep 11 14:19:38.188314 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.188199    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" event={"ID":"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6","Type":"ContainerDied","Data":"22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45"}
Sep 11 14:19:38.188314 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.188253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv" event={"ID":"6dfd9a8e-ecd0-467c-a382-5c2ee44124b6","Type":"ContainerDied","Data":"0c3c235e1a7a8587b44a181661825c2131bd99c35a4959740eb5c95d3e01b12d"}
Sep 11 14:19:38.188314 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.188274    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"
Sep 11 14:19:38.196698 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.196567    2558 scope.go:117] "RemoveContainer" containerID="d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520"
Sep 11 14:19:38.196942 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:19:38.196913    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520\": container with ID starting with d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520 not found: ID does not exist" containerID="d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520"
Sep 11 14:19:38.197048 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.196958    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520"} err="failed to get container status \"d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520\": rpc error: code = NotFound desc = could not find container \"d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520\": container with ID starting with d3af0f8e0b9685982a1cd659bd2154bb509ea47f5987bd3bd35475f206a5e520 not found: ID does not exist"
Sep 11 14:19:38.197048 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.196981    2558 scope.go:117] "RemoveContainer" containerID="22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45"
Sep 11 14:19:38.206960 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.206943    2558 scope.go:117] "RemoveContainer" containerID="22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45"
Sep 11 14:19:38.207296 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:19:38.207275    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45\": container with ID starting with 22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45 not found: ID does not exist" containerID="22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45"
Sep 11 14:19:38.207371 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.207306    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45"} err="failed to get container status \"22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45\": rpc error: code = NotFound desc = could not find container \"22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45\": container with ID starting with 22e747e5dcdf69104b1dd872690a794cc7b8b8cbfd2c6121e5cb80f8ccf00e45 not found: ID does not exist"
Sep 11 14:19:38.222752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.219730    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"]
Sep 11 14:19:38.226828 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.226800    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c6156e-68f5c4bd8b-wswgv"]
Sep 11 14:19:38.262017 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.261980    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"]
Sep 11 14:19:38.267600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.267569    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c6156e-6659fbbf66-jx9dk"]
Sep 11 14:19:38.416992 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.416956    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:38.504315 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.504279    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" path="/var/lib/kubelet/pods/6dfd9a8e-ecd0-467c-a382-5c2ee44124b6/volumes"
Sep 11 14:19:38.504763 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:38.504749    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" path="/var/lib/kubelet/pods/9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5/volumes"
Sep 11 14:19:39.037890 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:39.037835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:40.614566 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:40.614533    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:19:45.695206 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:45.695171    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:45.856513 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:45.856468    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:46.133896 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:46.133794    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:46.685690 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:46.685653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:47.176980 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:47.176940    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:47.428493 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:47.428406    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:48.189936 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:48.189902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:19:51.030950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:51.030916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:51.239083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:51.239047    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:53.031551 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:53.031515    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:53.166820 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:53.166785    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:53.347485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:53.347397    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:53.573128 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:53.573095    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:53.860814 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:53.860781    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:54.024624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.024588    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:54.285189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285159    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"]
Sep 11 14:19:54.285580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285484    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" containerName="payload-pre-processing"
Sep 11 14:19:54.285580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285495    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" containerName="payload-pre-processing"
Sep 11 14:19:54.285580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285510    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" containerName="maas-api"
Sep 11 14:19:54.285580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285517    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" containerName="maas-api"
Sep 11 14:19:54.285580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285576    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dfd9a8e-ecd0-467c-a382-5c2ee44124b6" containerName="maas-api"
Sep 11 14:19:54.285764 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.285587    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea6c311-fc5d-4ae5-ae2d-cde5a46715f5" containerName="payload-pre-processing"
Sep 11 14:19:54.290222 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.290205    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.295441 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.295417    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1e2e7d\""
Sep 11 14:19:54.295609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.295485    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1e2e7d-dockercfg-frcc6\""
Sep 11 14:19:54.307626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.307603    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"]
Sep 11 14:19:54.310746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.310724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.322249 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.322227    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"]
Sep 11 14:19:54.357764 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.357733    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"]
Sep 11 14:19:54.366285 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.366264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a186b216-4532-44a4-8f51-56ca6ccf4439-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.366391 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.366296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5rw\" (UniqueName: \"kubernetes.io/projected/8ca15502-7c36-491c-afab-96654299f89d-kube-api-access-4f5rw\") pod \"payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.366391 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.366317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989qf\" (UniqueName: \"kubernetes.io/projected/a186b216-4532-44a4-8f51-56ca6ccf4439-kube-api-access-989qf\") pod \"payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.366542 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.366512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca15502-7c36-491c-afab-96654299f89d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.467255 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.467223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a186b216-4532-44a4-8f51-56ca6ccf4439-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.467421 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.467261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5rw\" (UniqueName: \"kubernetes.io/projected/8ca15502-7c36-491c-afab-96654299f89d-kube-api-access-4f5rw\") pod \"payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.467421 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.467280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989qf\" (UniqueName: \"kubernetes.io/projected/a186b216-4532-44a4-8f51-56ca6ccf4439-kube-api-access-989qf\") pod \"payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.467421 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.467362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca15502-7c36-491c-afab-96654299f89d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.468052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.468005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a186b216-4532-44a4-8f51-56ca6ccf4439-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.468176 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.468102    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca15502-7c36-491c-afab-96654299f89d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.503267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.503241    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-989qf\" (UniqueName: \"kubernetes.io/projected/a186b216-4532-44a4-8f51-56ca6ccf4439-kube-api-access-989qf\") pod \"payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.511804 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.511781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5rw\" (UniqueName: \"kubernetes.io/projected/8ca15502-7c36-491c-afab-96654299f89d-kube-api-access-4f5rw\") pod \"payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.569013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.568923    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:54.599886 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.599855    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:54.619981 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.619954    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:54.849848 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.849760    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:54.890083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.888106    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"]
Sep 11 14:19:54.923005 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.922981    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"]
Sep 11 14:19:54.925544 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:19:54.925517    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca15502_7c36_491c_afab_96654299f89d.slice/crio-56ef09c747f8a67b1e8d0681b9ad82b49764197b13970ebf691c79d7cba6e744 WatchSource:0}: Error finding container 56ef09c747f8a67b1e8d0681b9ad82b49764197b13970ebf691c79d7cba6e744: Status 404 returned error can't find the container with id 56ef09c747f8a67b1e8d0681b9ad82b49764197b13970ebf691c79d7cba6e744
Sep 11 14:19:54.984658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:54.984627    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:55.274296 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.274260    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7" event={"ID":"8ca15502-7c36-491c-afab-96654299f89d","Type":"ContainerStarted","Data":"fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64"}
Sep 11 14:19:55.274501 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.274321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7" event={"ID":"8ca15502-7c36-491c-afab-96654299f89d","Type":"ContainerStarted","Data":"56ef09c747f8a67b1e8d0681b9ad82b49764197b13970ebf691c79d7cba6e744"}
Sep 11 14:19:55.274501 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.274375    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:19:55.276014 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.275986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c" event={"ID":"a186b216-4532-44a4-8f51-56ca6ccf4439","Type":"ContainerStarted","Data":"8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523"}
Sep 11 14:19:55.276156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.276058    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:19:55.276156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.276083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c" event={"ID":"a186b216-4532-44a4-8f51-56ca6ccf4439","Type":"ContainerStarted","Data":"4aad243ac8ef9f8e09239c1d4dce14a9ccc0463cadca0440c5728e7f37eeafde"}
Sep 11 14:19:55.296909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.296860    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7" podStartSLOduration=1.2968431040000001 podStartE2EDuration="1.296843104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:55.296715353 +0000 UTC m=+1329.260330333" watchObservedRunningTime="2026-09-11 14:19:55.296843104 +0000 UTC m=+1329.260458078"
Sep 11 14:19:55.317788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.317747    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c" podStartSLOduration=1.317732697 podStartE2EDuration="1.317732697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:55.317139319 +0000 UTC m=+1329.280754292" watchObservedRunningTime="2026-09-11 14:19:55.317732697 +0000 UTC m=+1329.281347671"
Sep 11 14:19:55.931797 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:55.931763    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:57.107765 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:57.107732    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:57.815190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:57.815156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:19:58.221332 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:58.221293    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:58.490844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:58.490756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:19:58.668074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:19:58.668014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:00.449724 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:00.449687    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:00.604249 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:00.604209    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:00.722955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:00.722870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:01.343854 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.343822    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:01.467730 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.467695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:01.709310 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.709278    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"]
Sep 11 14:20:01.713901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.713883    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:01.717056 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.717018    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-335a68\""
Sep 11 14:20:01.717163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.717112    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-335a68-dockercfg-lxvtm\""
Sep 11 14:20:01.736394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.736369    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"]
Sep 11 14:20:01.834045 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.833993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7946ec-d010-4180-8246-1245e5c6393a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:01.834202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.834096    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nshwz\" (UniqueName: \"kubernetes.io/projected/cc7946ec-d010-4180-8246-1245e5c6393a-kube-api-access-nshwz\") pod \"payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:01.934697 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.934648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7946ec-d010-4180-8246-1245e5c6393a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:01.934904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.934717    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nshwz\" (UniqueName: \"kubernetes.io/projected/cc7946ec-d010-4180-8246-1245e5c6393a-kube-api-access-nshwz\") pod \"payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:01.935362 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.935340    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7946ec-d010-4180-8246-1245e5c6393a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:01.945894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:01.945865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nshwz\" (UniqueName: \"kubernetes.io/projected/cc7946ec-d010-4180-8246-1245e5c6393a-kube-api-access-nshwz\") pod \"payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:02.023858 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:02.023761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:02.319763 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:02.319738    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"]
Sep 11 14:20:02.322313 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:02.322285    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7946ec_d010_4180_8246_1245e5c6393a.slice/crio-1ad8f18229c0c04a376e7f7574b03ef6557ed7ce7801dede4a204f32bbe06e68 WatchSource:0}: Error finding container 1ad8f18229c0c04a376e7f7574b03ef6557ed7ce7801dede4a204f32bbe06e68: Status 404 returned error can't find the container with id 1ad8f18229c0c04a376e7f7574b03ef6557ed7ce7801dede4a204f32bbe06e68
Sep 11 14:20:02.333647 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:02.333621    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:03.317189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:03.317153    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq" event={"ID":"cc7946ec-d010-4180-8246-1245e5c6393a","Type":"ContainerStarted","Data":"33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963"}
Sep 11 14:20:03.317610 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:03.317207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq" event={"ID":"cc7946ec-d010-4180-8246-1245e5c6393a","Type":"ContainerStarted","Data":"1ad8f18229c0c04a376e7f7574b03ef6557ed7ce7801dede4a204f32bbe06e68"}
Sep 11 14:20:03.317610 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:03.317282    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:03.341708 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:03.341657    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq" podStartSLOduration=2.341639245 podStartE2EDuration="2.341639245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:03.340342953 +0000 UTC m=+1337.303957928" watchObservedRunningTime="2026-09-11 14:20:03.341639245 +0000 UTC m=+1337.305254282"
Sep 11 14:20:03.504300 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:03.504269    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:04.391920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:04.391883    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:05.233864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:05.233827    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:05.601310 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:05.601207    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:05.981584 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:05.981550    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:06.283656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:06.283573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:20:06.283656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:06.283643    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:20:06.429840 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:06.429807    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:06.660221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:06.660135    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:06.857156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:06.857120    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:07.868954 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:07.868916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:09.496290 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:09.496215    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:10.118161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:10.118119    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:10.245464 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:10.245435    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:10.796970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:10.796917    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:11.963866 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:11.963821    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:12.896362 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:12.896325    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:13.319157 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.319125    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:13.723965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.723926    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-beee0a"]
Sep 11 14:20:13.728596 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.728572    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:13.731528 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.731508    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4lz74\""
Sep 11 14:20:13.737539 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.737517    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-beee0a"]
Sep 11 14:20:13.744534 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.744513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zccl\" (UniqueName: \"kubernetes.io/projected/4a6f564b-35b8-497c-a0f1-daeda71afe07-kube-api-access-8zccl\") pod \"test-curl-7705-beee0a\" (UID: \"4a6f564b-35b8-497c-a0f1-daeda71afe07\") " pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:13.845661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.845621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zccl\" (UniqueName: \"kubernetes.io/projected/4a6f564b-35b8-497c-a0f1-daeda71afe07-kube-api-access-8zccl\") pod \"test-curl-7705-beee0a\" (UID: \"4a6f564b-35b8-497c-a0f1-daeda71afe07\") " pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:13.856363 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:13.856333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zccl\" (UniqueName: \"kubernetes.io/projected/4a6f564b-35b8-497c-a0f1-daeda71afe07-kube-api-access-8zccl\") pod \"test-curl-7705-beee0a\" (UID: \"4a6f564b-35b8-497c-a0f1-daeda71afe07\") " pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:14.038174 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:14.038081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:14.223637 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:14.223603    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:14.311658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:14.311624    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-beee0a"]
Sep 11 14:20:14.314089 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:14.314058    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a6f564b_35b8_497c_a0f1_daeda71afe07.slice/crio-a1c252eb30cbd5c1c575f939d29d06c91b25c7541057558054fd1386c8e17e58 WatchSource:0}: Error finding container a1c252eb30cbd5c1c575f939d29d06c91b25c7541057558054fd1386c8e17e58: Status 404 returned error can't find the container with id a1c252eb30cbd5c1c575f939d29d06c91b25c7541057558054fd1386c8e17e58
Sep 11 14:20:14.325330 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:14.325305    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:20:14.370094 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:14.370068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-beee0a" event={"ID":"4a6f564b-35b8-497c-a0f1-daeda71afe07","Type":"ContainerStarted","Data":"a1c252eb30cbd5c1c575f939d29d06c91b25c7541057558054fd1386c8e17e58"}
Sep 11 14:20:15.375795 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:15.375753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-beee0a" event={"ID":"4a6f564b-35b8-497c-a0f1-daeda71afe07","Type":"ContainerStarted","Data":"c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8"}
Sep 11 14:20:15.406208 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:15.406157    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-beee0a" podStartSLOduration=2.406140668 podStartE2EDuration="2.406140668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:15.405242377 +0000 UTC m=+1349.368857345" watchObservedRunningTime="2026-09-11 14:20:15.406140668 +0000 UTC m=+1349.369755690"
Sep 11 14:20:15.601580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:15.601547    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:15.710911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:15.710863    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:16.118001 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.117914    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-beee0a"]
Sep 11 14:20:16.124932 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.124894    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-beee0a"]
Sep 11 14:20:16.156639 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.156601    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:16.264732 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.264694    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:16.265561 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.265537    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-729bf1"]
Sep 11 14:20:16.265944 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.265929    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6f564b-35b8-497c-a0f1-daeda71afe07" containerName="test-curl-7705-beee0a"
Sep 11 14:20:16.265993 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.265946    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6f564b-35b8-497c-a0f1-daeda71afe07" containerName="test-curl-7705-beee0a"
Sep 11 14:20:16.266042 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.266012    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6f564b-35b8-497c-a0f1-daeda71afe07" containerName="test-curl-7705-beee0a"
Sep 11 14:20:16.268925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.268906    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:16.275249 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.275209    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"]
Sep 11 14:20:16.278905 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.278878    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-729bf1"]
Sep 11 14:20:16.279061 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.278986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.282122 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.282099    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-031a56\""
Sep 11 14:20:16.296978 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.296949    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"]
Sep 11 14:20:16.365801 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.365750    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9f040-38a5-4367-9bb1-678e7324fc5c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.365801 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.365805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsv6\" (UniqueName: \"kubernetes.io/projected/79b9f040-38a5-4367-9bb1-678e7324fc5c-kube-api-access-fbsv6\") pod \"payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.366065 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.365853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqp8d\" (UniqueName: \"kubernetes.io/projected/8596c458-0528-4811-a809-866410761712-kube-api-access-rqp8d\") pod \"test-curl-7705-729bf1\" (UID: \"8596c458-0528-4811-a809-866410761712\") " pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:16.380827 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:16.380737    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-beee0a" container="test-curl-7705-beee0a"
Sep 11 14:20:16.467170 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.467127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqp8d\" (UniqueName: \"kubernetes.io/projected/8596c458-0528-4811-a809-866410761712-kube-api-access-rqp8d\") pod \"test-curl-7705-729bf1\" (UID: \"8596c458-0528-4811-a809-866410761712\") " pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:16.467364 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.467230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9f040-38a5-4367-9bb1-678e7324fc5c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.467364 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.467261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsv6\" (UniqueName: \"kubernetes.io/projected/79b9f040-38a5-4367-9bb1-678e7324fc5c-kube-api-access-fbsv6\") pod \"payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.468048 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.468007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9f040-38a5-4367-9bb1-678e7324fc5c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.478839 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.478805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqp8d\" (UniqueName: \"kubernetes.io/projected/8596c458-0528-4811-a809-866410761712-kube-api-access-rqp8d\") pod \"test-curl-7705-729bf1\" (UID: \"8596c458-0528-4811-a809-866410761712\") " pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:16.479071 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.479049    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsv6\" (UniqueName: \"kubernetes.io/projected/79b9f040-38a5-4367-9bb1-678e7324fc5c-kube-api-access-fbsv6\") pod \"payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.578947 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.578911    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:16.579620 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.579601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:16.597001 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.596970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:16.677907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.677862    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:16.859689 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.859660    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-729bf1"]
Sep 11 14:20:16.861808 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:16.861781    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8596c458_0528_4811_a809_866410761712.slice/crio-211a220a44a0d4e733d17f975a86d88d1bcae81000bb4cb522b18abb43687225 WatchSource:0}: Error finding container 211a220a44a0d4e733d17f975a86d88d1bcae81000bb4cb522b18abb43687225: Status 404 returned error can't find the container with id 211a220a44a0d4e733d17f975a86d88d1bcae81000bb4cb522b18abb43687225
Sep 11 14:20:16.883720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:16.883603    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"]
Sep 11 14:20:16.894598 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:16.894540    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b9f040_38a5_4367_9bb1_678e7324fc5c.slice/crio-e5edcbdd28f545be390ff47dcd90190ee853cfb10efebc5ec7d4a1d9bcaadbc2 WatchSource:0}: Error finding container e5edcbdd28f545be390ff47dcd90190ee853cfb10efebc5ec7d4a1d9bcaadbc2: Status 404 returned error can't find the container with id e5edcbdd28f545be390ff47dcd90190ee853cfb10efebc5ec7d4a1d9bcaadbc2
Sep 11 14:20:17.121799 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.121764    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:20:17.390346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.390308    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-729bf1" event={"ID":"8596c458-0528-4811-a809-866410761712","Type":"ContainerStarted","Data":"0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167"}
Sep 11 14:20:17.390833 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.390356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-729bf1" event={"ID":"8596c458-0528-4811-a809-866410761712","Type":"ContainerStarted","Data":"211a220a44a0d4e733d17f975a86d88d1bcae81000bb4cb522b18abb43687225"}
Sep 11 14:20:17.391967 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.391935    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff" event={"ID":"79b9f040-38a5-4367-9bb1-678e7324fc5c","Type":"ContainerStarted","Data":"05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4"}
Sep 11 14:20:17.392124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.391997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:17.392124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.392023    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff" event={"ID":"79b9f040-38a5-4367-9bb1-678e7324fc5c","Type":"ContainerStarted","Data":"e5edcbdd28f545be390ff47dcd90190ee853cfb10efebc5ec7d4a1d9bcaadbc2"}
Sep 11 14:20:17.392124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.392052    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-beee0a" podUID="4a6f564b-35b8-497c-a0f1-daeda71afe07" containerName="test-curl-7705-beee0a" containerID="cri-o://c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8" gracePeriod=2
Sep 11 14:20:17.412341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.412298    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-729bf1" podStartSLOduration=1.412282977 podStartE2EDuration="1.412282977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:17.411839004 +0000 UTC m=+1351.375453969" watchObservedRunningTime="2026-09-11 14:20:17.412282977 +0000 UTC m=+1351.375897951"
Sep 11 14:20:17.435789 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.435746    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff" podStartSLOduration=1.435730106 podStartE2EDuration="1.435730106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:17.434349144 +0000 UTC m=+1351.397964117" watchObservedRunningTime="2026-09-11 14:20:17.435730106 +0000 UTC m=+1351.399345120"
Sep 11 14:20:17.869778 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:17.869736    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:18.138069 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.137957    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-729bf1"]
Sep 11 14:20:18.140574 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.140535    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-729bf1"]
Sep 11 14:20:18.267287 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.267253    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-7f8a86"]
Sep 11 14:20:18.267634 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.267621    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8596c458-0528-4811-a809-866410761712" containerName="test-curl-7705-729bf1"
Sep 11 14:20:18.267680 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.267635    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8596c458-0528-4811-a809-866410761712" containerName="test-curl-7705-729bf1"
Sep 11 14:20:18.267719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.267711    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8596c458-0528-4811-a809-866410761712" containerName="test-curl-7705-729bf1"
Sep 11 14:20:18.270890 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.270871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:18.283963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.283929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnz5\" (UniqueName: \"kubernetes.io/projected/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53-kube-api-access-lqnz5\") pod \"test-curl-7705-7f8a86\" (UID: \"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53\") " pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:18.284351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.284325    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-7f8a86"]
Sep 11 14:20:18.385350 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.385311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnz5\" (UniqueName: \"kubernetes.io/projected/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53-kube-api-access-lqnz5\") pod \"test-curl-7705-7f8a86\" (UID: \"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53\") " pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:18.394907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.394825    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnz5\" (UniqueName: \"kubernetes.io/projected/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53-kube-api-access-lqnz5\") pod \"test-curl-7705-7f8a86\" (UID: \"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53\") " pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:18.397648 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:18.397615    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-729bf1" container="test-curl-7705-729bf1"
Sep 11 14:20:18.583995 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.583957    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:18.866082 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.866053    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-7f8a86"]
Sep 11 14:20:18.868433 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:18.868403    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e4d17f6_4b97_4816_b1fc_7c4f218a5a53.slice/crio-618ed7c960e44a86a0f5339beae40aa14b898198502f67fb2322f91a4258d19b WatchSource:0}: Error finding container 618ed7c960e44a86a0f5339beae40aa14b898198502f67fb2322f91a4258d19b: Status 404 returned error can't find the container with id 618ed7c960e44a86a0f5339beae40aa14b898198502f67fb2322f91a4258d19b
Sep 11 14:20:18.879871 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:18.879847    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:19.403178 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.403146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-7f8a86" event={"ID":"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53","Type":"ContainerStarted","Data":"7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d"}
Sep 11 14:20:19.403536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.403181    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-729bf1" podUID="8596c458-0528-4811-a809-866410761712" containerName="test-curl-7705-729bf1" containerID="cri-o://0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167" gracePeriod=2
Sep 11 14:20:19.403536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.403194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-7f8a86" event={"ID":"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53","Type":"ContainerStarted","Data":"618ed7c960e44a86a0f5339beae40aa14b898198502f67fb2322f91a4258d19b"}
Sep 11 14:20:19.423846 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.423798    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-7f8a86" podStartSLOduration=1.423782761 podStartE2EDuration="1.423782761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:19.42187871 +0000 UTC m=+1353.385493679" watchObservedRunningTime="2026-09-11 14:20:19.423782761 +0000 UTC m=+1353.387397779"
Sep 11 14:20:19.566300 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.566272    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:19.594915 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.594892    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zccl\" (UniqueName: \"kubernetes.io/projected/4a6f564b-35b8-497c-a0f1-daeda71afe07-kube-api-access-8zccl\") pod \"4a6f564b-35b8-497c-a0f1-daeda71afe07\" (UID: \"4a6f564b-35b8-497c-a0f1-daeda71afe07\") "
Sep 11 14:20:19.596901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.596877    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6f564b-35b8-497c-a0f1-daeda71afe07-kube-api-access-8zccl" (OuterVolumeSpecName: "kube-api-access-8zccl") pod "4a6f564b-35b8-497c-a0f1-daeda71afe07" (UID: "4a6f564b-35b8-497c-a0f1-daeda71afe07"). InnerVolumeSpecName "kube-api-access-8zccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:19.696038 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:19.695997    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zccl\" (UniqueName: \"kubernetes.io/projected/4a6f564b-35b8-497c-a0f1-daeda71afe07-kube-api-access-8zccl\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:20:20.099248 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.099163    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-7f8a86"]
Sep 11 14:20:20.104570 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.104543    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-7f8a86"]
Sep 11 14:20:20.237787 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.237754    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-4e049b"]
Sep 11 14:20:20.238181 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.238168    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" containerName="test-curl-7705-7f8a86"
Sep 11 14:20:20.238234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.238183    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" containerName="test-curl-7705-7f8a86"
Sep 11 14:20:20.238271 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.238240    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" containerName="test-curl-7705-7f8a86"
Sep 11 14:20:20.241251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.241230    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:20.248507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.248441    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-4e049b"]
Sep 11 14:20:20.300707 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.300671    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2g5g\" (UniqueName: \"kubernetes.io/projected/4a327fa1-dc36-49a7-9b1a-c459c21166af-kube-api-access-w2g5g\") pod \"test-curl-7705-4e049b\" (UID: \"4a327fa1-dc36-49a7-9b1a-c459c21166af\") " pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:20.401344 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.401259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2g5g\" (UniqueName: \"kubernetes.io/projected/4a327fa1-dc36-49a7-9b1a-c459c21166af-kube-api-access-w2g5g\") pod \"test-curl-7705-4e049b\" (UID: \"4a327fa1-dc36-49a7-9b1a-c459c21166af\") " pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:20.409350 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.409314    2558 generic.go:356] "Generic (PLEG): container finished" podID="4a6f564b-35b8-497c-a0f1-daeda71afe07" containerID="c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8" exitCode=137
Sep 11 14:20:20.409773 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.409377    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:20.409773 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.409414    2558 scope.go:117] "RemoveContainer" containerID="c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8"
Sep 11 14:20:20.409773 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:20.409521    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-7f8a86" container="test-curl-7705-7f8a86"
Sep 11 14:20:20.411607 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.411586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2g5g\" (UniqueName: \"kubernetes.io/projected/4a327fa1-dc36-49a7-9b1a-c459c21166af-kube-api-access-w2g5g\") pod \"test-curl-7705-4e049b\" (UID: \"4a327fa1-dc36-49a7-9b1a-c459c21166af\") " pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:20.426607 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:20.426575    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-beee0a\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="4a6f564b-35b8-497c-a0f1-daeda71afe07" pod="opendatahub/test-curl-7705-beee0a"
Sep 11 14:20:20.428470 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.428453    2558 scope.go:117] "RemoveContainer" containerID="c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8"
Sep 11 14:20:20.428750 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:20.428728    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8\": container with ID starting with c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8 not found: ID does not exist" containerID="c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8"
Sep 11 14:20:20.428814 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.428757    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8"} err="failed to get container status \"c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8\": rpc error: code = NotFound desc = could not find container \"c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8\": container with ID starting with c517c71cad6b3e55b986be7e93904e374fceb4ea8f295b2babed3dbde7aa0da8 not found: ID does not exist"
Sep 11 14:20:20.504879 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.504848    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6f564b-35b8-497c-a0f1-daeda71afe07" path="/var/lib/kubelet/pods/4a6f564b-35b8-497c-a0f1-daeda71afe07/volumes"
Sep 11 14:20:20.550922 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.550890    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:20.826658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:20.826624    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-4e049b"]
Sep 11 14:20:20.829642 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:20.829609    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a327fa1_dc36_49a7_9b1a_c459c21166af.slice/crio-eb5eed40612303f8cf9812d80c84d7ab5177fddd3d7a41ed25b6f439c2a3f419 WatchSource:0}: Error finding container eb5eed40612303f8cf9812d80c84d7ab5177fddd3d7a41ed25b6f439c2a3f419: Status 404 returned error can't find the container with id eb5eed40612303f8cf9812d80c84d7ab5177fddd3d7a41ed25b6f439c2a3f419
Sep 11 14:20:21.416211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.416161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-4e049b" event={"ID":"4a327fa1-dc36-49a7-9b1a-c459c21166af","Type":"ContainerStarted","Data":"4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d"}
Sep 11 14:20:21.416211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.416209    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-4e049b" event={"ID":"4a327fa1-dc36-49a7-9b1a-c459c21166af","Type":"ContainerStarted","Data":"eb5eed40612303f8cf9812d80c84d7ab5177fddd3d7a41ed25b6f439c2a3f419"}
Sep 11 14:20:21.417286 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.417244    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-7f8a86" podUID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" containerName="test-curl-7705-7f8a86" containerID="cri-o://7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d" gracePeriod=2
Sep 11 14:20:21.449645 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.449565    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-4e049b" podStartSLOduration=1.449550069 podStartE2EDuration="1.449550069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:21.447325401 +0000 UTC m=+1355.410940375" watchObservedRunningTime="2026-09-11 14:20:21.449550069 +0000 UTC m=+1355.413165042"
Sep 11 14:20:21.574136 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.574107    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:21.612427 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.612404    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqp8d\" (UniqueName: \"kubernetes.io/projected/8596c458-0528-4811-a809-866410761712-kube-api-access-rqp8d\") pod \"8596c458-0528-4811-a809-866410761712\" (UID: \"8596c458-0528-4811-a809-866410761712\") "
Sep 11 14:20:21.614689 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.614665    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8596c458-0528-4811-a809-866410761712-kube-api-access-rqp8d" (OuterVolumeSpecName: "kube-api-access-rqp8d") pod "8596c458-0528-4811-a809-866410761712" (UID: "8596c458-0528-4811-a809-866410761712"). InnerVolumeSpecName "kube-api-access-rqp8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:21.713832 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:21.713806    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqp8d\" (UniqueName: \"kubernetes.io/projected/8596c458-0528-4811-a809-866410761712-kube-api-access-rqp8d\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:20:22.226765 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.226727    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:22.256657 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.256625    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-4e049b"]
Sep 11 14:20:22.260296 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.260271    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-4e049b"]
Sep 11 14:20:22.423180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.423143    2558 generic.go:356] "Generic (PLEG): container finished" podID="8596c458-0528-4811-a809-866410761712" containerID="0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167" exitCode=137
Sep 11 14:20:22.423578 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.423183    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:22.423578 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.423242    2558 scope.go:117] "RemoveContainer" containerID="0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167"
Sep 11 14:20:22.423578 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:22.423525    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-4e049b" container="test-curl-7705-4e049b"
Sep 11 14:20:22.431864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.431849    2558 scope.go:117] "RemoveContainer" containerID="0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167"
Sep 11 14:20:22.432160 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:22.432137    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167\": container with ID starting with 0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167 not found: ID does not exist" containerID="0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167"
Sep 11 14:20:22.432202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.432170    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167"} err="failed to get container status \"0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167\": rpc error: code = NotFound desc = could not find container \"0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167\": container with ID starting with 0cc3c1fab5492fd5a4eeed7303bfe432a1ce1d7617885adc0fa1d7ebc96aa167 not found: ID does not exist"
Sep 11 14:20:22.433360 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:22.433338    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-729bf1\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="8596c458-0528-4811-a809-866410761712" pod="opendatahub/test-curl-7705-729bf1"
Sep 11 14:20:22.503331 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.503261    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8596c458-0528-4811-a809-866410761712" path="/var/lib/kubelet/pods/8596c458-0528-4811-a809-866410761712/volumes"
Sep 11 14:20:22.971255 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:22.971218    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:23.384286 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.384189    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:23.428693 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.428662    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-4e049b" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" containerName="test-curl-7705-4e049b" containerID="cri-o://4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d" gracePeriod=2
Sep 11 14:20:23.499121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.499094    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:23.609390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.609362    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:23.629538 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.629507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqnz5\" (UniqueName: \"kubernetes.io/projected/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53-kube-api-access-lqnz5\") pod \"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53\" (UID: \"2e4d17f6-4b97-4816-b1fc-7c4f218a5a53\") "
Sep 11 14:20:23.631417 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.631389    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53-kube-api-access-lqnz5" (OuterVolumeSpecName: "kube-api-access-lqnz5") pod "2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" (UID: "2e4d17f6-4b97-4816-b1fc-7c4f218a5a53"). InnerVolumeSpecName "kube-api-access-lqnz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:23.730722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:23.730681    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnz5\" (UniqueName: \"kubernetes.io/projected/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53-kube-api-access-lqnz5\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:20:24.434579 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:24.434544    2558 generic.go:356] "Generic (PLEG): container finished" podID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" containerID="7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d" exitCode=137
Sep 11 14:20:24.434579 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:24.434583    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:24.435167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:24.434646    2558 scope.go:117] "RemoveContainer" containerID="7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d"
Sep 11 14:20:24.443267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:24.443253    2558 scope.go:117] "RemoveContainer" containerID="7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d"
Sep 11 14:20:24.443529 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:24.443514    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d\": container with ID starting with 7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d not found: ID does not exist" containerID="7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d"
Sep 11 14:20:24.443568 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:24.443536    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d"} err="failed to get container status \"7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d\": rpc error: code = NotFound desc = could not find container \"7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d\": container with ID starting with 7841acfb3b011e5f381a07abaeb1b846ad49cf185cedda1fd7abcf42b678572d not found: ID does not exist"
Sep 11 14:20:24.446582 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:24.446557    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-7f8a86\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" pod="opendatahub/test-curl-7705-7f8a86"
Sep 11 14:20:24.502871 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:24.502844    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e4d17f6-4b97-4816-b1fc-7c4f218a5a53" path="/var/lib/kubelet/pods/2e4d17f6-4b97-4816-b1fc-7c4f218a5a53/volumes"
Sep 11 14:20:25.460660 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:25.460632    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:25.614623 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:25.614601    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:25.648763 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:25.648737    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2g5g\" (UniqueName: \"kubernetes.io/projected/4a327fa1-dc36-49a7-9b1a-c459c21166af-kube-api-access-w2g5g\") pod \"4a327fa1-dc36-49a7-9b1a-c459c21166af\" (UID: \"4a327fa1-dc36-49a7-9b1a-c459c21166af\") "
Sep 11 14:20:25.650712 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:25.650690    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a327fa1-dc36-49a7-9b1a-c459c21166af-kube-api-access-w2g5g" (OuterVolumeSpecName: "kube-api-access-w2g5g") pod "4a327fa1-dc36-49a7-9b1a-c459c21166af" (UID: "4a327fa1-dc36-49a7-9b1a-c459c21166af"). InnerVolumeSpecName "kube-api-access-w2g5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:25.750359 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:25.750307    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2g5g\" (UniqueName: \"kubernetes.io/projected/4a327fa1-dc36-49a7-9b1a-c459c21166af-kube-api-access-w2g5g\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:20:26.380478 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.380437    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:26.446844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.446805    2558 generic.go:356] "Generic (PLEG): container finished" podID="4a327fa1-dc36-49a7-9b1a-c459c21166af" containerID="4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d" exitCode=137
Sep 11 14:20:26.446844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.446844    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:26.447124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.446888    2558 scope.go:117] "RemoveContainer" containerID="4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d"
Sep 11 14:20:26.455514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.455496    2558 scope.go:117] "RemoveContainer" containerID="4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d"
Sep 11 14:20:26.455802 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:26.455783    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d\": container with ID starting with 4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d not found: ID does not exist" containerID="4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d"
Sep 11 14:20:26.455853 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.455812    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d"} err="failed to get container status \"4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d\": rpc error: code = NotFound desc = could not find container \"4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d\": container with ID starting with 4a7477a2ac5eab1b6e7d592a75489dbe958266d17d5b4ece7cc04885428e023d not found: ID does not exist"
Sep 11 14:20:26.457574 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:26.457552    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-4e049b\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:26.502483 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:26.502453    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-4e049b\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" pod="opendatahub/test-curl-7705-4e049b"
Sep 11 14:20:26.503440 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:26.503419    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" path="/var/lib/kubelet/pods/4a327fa1-dc36-49a7-9b1a-c459c21166af/volumes"
Sep 11 14:20:27.168052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:27.167994    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:27.299185 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:27.299146    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:27.641926 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:27.641889    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:28.399345 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:28.399315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:20:28.751070 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:28.751022    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:29.649938 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:29.649901    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:30.832906 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:30.832868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:31.725876 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.725833    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"]
Sep 11 14:20:31.726330 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.726310    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" containerName="test-curl-7705-4e049b"
Sep 11 14:20:31.726392 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.726332    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" containerName="test-curl-7705-4e049b"
Sep 11 14:20:31.726435 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.726406    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a327fa1-dc36-49a7-9b1a-c459c21166af" containerName="test-curl-7705-4e049b"
Sep 11 14:20:31.731334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.731315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.734143 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.734119    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-557930d3-serving-cert\""
Sep 11 14:20:31.734265 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.734126    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktm9c\""
Sep 11 14:20:31.735102 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.735085    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:20:31.735216 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.735104    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:20:31.735216 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.735145    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:20:31.743893 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.743868    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"]
Sep 11 14:20:31.777474 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.777432    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:31.804978 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.804947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.805122 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.804984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-metrics-certs\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.805173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.805127    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcch9\" (UniqueName: \"kubernetes.io/projected/d5beed71-bb68-482f-b7f8-b3cb4fee462b-kube-api-access-pcch9\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.805173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.805161    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5beed71-bb68-482f-b7f8-b3cb4fee462b-openshift-service-ca\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.873385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.873364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:31.906421 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.906394    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcch9\" (UniqueName: \"kubernetes.io/projected/d5beed71-bb68-482f-b7f8-b3cb4fee462b-kube-api-access-pcch9\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.906578 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.906433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5beed71-bb68-482f-b7f8-b3cb4fee462b-openshift-service-ca\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.906578 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.906488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.906578 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.906509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-metrics-certs\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.906738 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:31.906610    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-serving-cert: secret "maas-api-e2e-mt-557930d3-serving-cert" not found
Sep 11 14:20:31.906738 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:20:31.906663    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls podName:d5beed71-bb68-482f-b7f8-b3cb4fee462b nodeName:}" failed. No retries permitted until 2026-09-11 14:20:32.406648781 +0000 UTC m=+1366.370263738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls") pod "maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" (UID: "d5beed71-bb68-482f-b7f8-b3cb4fee462b") : secret "maas-api-e2e-mt-557930d3-serving-cert" not found
Sep 11 14:20:31.907160 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.907141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5beed71-bb68-482f-b7f8-b3cb4fee462b-openshift-service-ca\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.908827 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.908808    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-metrics-certs\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:31.918121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:31.918101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcch9\" (UniqueName: \"kubernetes.io/projected/d5beed71-bb68-482f-b7f8-b3cb4fee462b-kube-api-access-pcch9\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:32.369407 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:32.369368    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:32.410666 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:32.410626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:32.413526 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:32.413492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls\") pod \"maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:32.482728 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:32.482692    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:32.643245 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:32.643164    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:32.742423 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:32.742386    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:33.006246 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.006215    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"]
Sep 11 14:20:33.010047 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:33.010001    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5beed71_bb68_482f_b7f8_b3cb4fee462b.slice/crio-9410940ec5078e2066791fe98b3188e712bd8b4d7246ea43985a6509dbba9ec8 WatchSource:0}: Error finding container 9410940ec5078e2066791fe98b3188e712bd8b4d7246ea43985a6509dbba9ec8: Status 404 returned error can't find the container with id 9410940ec5078e2066791fe98b3188e712bd8b4d7246ea43985a6509dbba9ec8
Sep 11 14:20:33.137747 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.137717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"]
Sep 11 14:20:33.142479 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.142454    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.145467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.145444    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-557930d3\""
Sep 11 14:20:33.147353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.147330    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-557930d3-dockercfg-l288l\""
Sep 11 14:20:33.158753 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.158729    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"]
Sep 11 14:20:33.176945 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.176911    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:20:33.217922 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.217900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srdc\" (UniqueName: \"kubernetes.io/projected/bd36df79-c545-46ef-a01d-bdeaf1d228aa-kube-api-access-8srdc\") pod \"payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") " pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.218062 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.217964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36df79-c545-46ef-a01d-bdeaf1d228aa-plugins-config-volume\") pod \"payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") " pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.319458 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.319377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srdc\" (UniqueName: \"kubernetes.io/projected/bd36df79-c545-46ef-a01d-bdeaf1d228aa-kube-api-access-8srdc\") pod \"payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") " pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.319458 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.319447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36df79-c545-46ef-a01d-bdeaf1d228aa-plugins-config-volume\") pod \"payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") " pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.320158 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.320140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36df79-c545-46ef-a01d-bdeaf1d228aa-plugins-config-volume\") pod \"payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") " pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.328713 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.328686    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srdc\" (UniqueName: \"kubernetes.io/projected/bd36df79-c545-46ef-a01d-bdeaf1d228aa-kube-api-access-8srdc\") pod \"payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") " pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.425571 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.425531    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:33.453420 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.453392    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:33.482565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.482474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" event={"ID":"d5beed71-bb68-482f-b7f8-b3cb4fee462b","Type":"ContainerStarted","Data":"9410940ec5078e2066791fe98b3188e712bd8b4d7246ea43985a6509dbba9ec8"}
Sep 11 14:20:33.733905 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:33.733867    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"]
Sep 11 14:20:33.735740 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:33.735714    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd36df79_c545_46ef_a01d_bdeaf1d228aa.slice/crio-3e7aecbea2878f3535e85c971254eaa728580fbb67ca7a7988a3c4dda89109fc WatchSource:0}: Error finding container 3e7aecbea2878f3535e85c971254eaa728580fbb67ca7a7988a3c4dda89109fc: Status 404 returned error can't find the container with id 3e7aecbea2878f3535e85c971254eaa728580fbb67ca7a7988a3c4dda89109fc
Sep 11 14:20:34.448810 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.448771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:34.489636 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.489603    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4" event={"ID":"bd36df79-c545-46ef-a01d-bdeaf1d228aa","Type":"ContainerStarted","Data":"85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc"}
Sep 11 14:20:34.489829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.489695    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:34.489829 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.489721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4" event={"ID":"bd36df79-c545-46ef-a01d-bdeaf1d228aa","Type":"ContainerStarted","Data":"3e7aecbea2878f3535e85c971254eaa728580fbb67ca7a7988a3c4dda89109fc"}
Sep 11 14:20:34.491373 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.491346    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" event={"ID":"d5beed71-bb68-482f-b7f8-b3cb4fee462b","Type":"ContainerStarted","Data":"ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5"}
Sep 11 14:20:34.491499 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.491483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:34.518185 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.518141    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4" podStartSLOduration=1.51812597 podStartE2EDuration="1.51812597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:34.516563557 +0000 UTC m=+1368.480178532" watchObservedRunningTime="2026-09-11 14:20:34.51812597 +0000 UTC m=+1368.481740944"
Sep 11 14:20:34.538540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:34.538486    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" podStartSLOduration=3.083325352 podStartE2EDuration="3.538467547s" totalImagesPullingTime="455.142195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:34.536768111 +0000 UTC m=+1368.500383084" watchObservedRunningTime="2026-09-11 14:20:34.538467547 +0000 UTC m=+1368.502082525"
Sep 11 14:20:35.125270 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:35.125221    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:35.265159 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:35.265126    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:36.073768 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:36.073729    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:36.976662 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:36.976627    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:37.828683 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:37.828653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:38.797389 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:38.797356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:40.505178 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:40.505152    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:20:45.502085 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:45.502051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:20:48.079311 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:48.079273    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:20:49.854018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.853983    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"]
Sep 11 14:20:49.858727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.858709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:49.861454 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.861429    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"openshift-service-ca.crt\""
Sep 11 14:20:49.861575 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.861456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"default-dockercfg-zvrmr\""
Sep 11 14:20:49.862471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.862456    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"kube-root-ca.crt\""
Sep 11 14:20:49.862529 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.862501    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"auth-test-model-1e2e7d-kserve-self-signed-certs\""
Sep 11 14:20:49.868427 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.868402    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"]
Sep 11 14:20:49.971434 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.971399    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:49.977403 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.977374    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-home\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:49.977546 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.977416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fe0-1e45-43d0-aa60-23ddc24d372f-tls-certs\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:49.977546 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.977439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-dshm\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:49.977546 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.977454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-model-cache\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:49.977664 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.977548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlljw\" (UniqueName: \"kubernetes.io/projected/46254fe0-1e45-43d0-aa60-23ddc24d372f-kube-api-access-mlljw\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:49.977664 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:49.977632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-tmp-dir\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078459 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-tmp-dir\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078459 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-home\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fe0-1e45-43d0-aa60-23ddc24d372f-tls-certs\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-dshm\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-model-cache\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078841 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlljw\" (UniqueName: \"kubernetes.io/projected/46254fe0-1e45-43d0-aa60-23ddc24d372f-kube-api-access-mlljw\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078899 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078867    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-home\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.078970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.078949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-tmp-dir\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.079060 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.079021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-model-cache\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.080867 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.080838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-dshm\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.081091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.081076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fe0-1e45-43d0-aa60-23ddc24d372f-tls-certs\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.089901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.089876    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlljw\" (UniqueName: \"kubernetes.io/projected/46254fe0-1e45-43d0-aa60-23ddc24d372f-kube-api-access-mlljw\") pod \"auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.169134 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.169014    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:20:50.443767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.443743    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"]
Sep 11 14:20:50.445416 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:20:50.445393    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46254fe0_1e45_43d0_aa60_23ddc24d372f.slice/crio-9d65d4a5e48e64ebe9800a3061d976763de874417929ec03163bb2030c5d61e3 WatchSource:0}: Error finding container 9d65d4a5e48e64ebe9800a3061d976763de874417929ec03163bb2030c5d61e3: Status 404 returned error can't find the container with id 9d65d4a5e48e64ebe9800a3061d976763de874417929ec03163bb2030c5d61e3
Sep 11 14:20:50.576144 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.576103    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br" event={"ID":"46254fe0-1e45-43d0-aa60-23ddc24d372f","Type":"ContainerStarted","Data":"2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c"}
Sep 11 14:20:50.576302 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.576155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br" event={"ID":"46254fe0-1e45-43d0-aa60-23ddc24d372f","Type":"ContainerStarted","Data":"9d65d4a5e48e64ebe9800a3061d976763de874417929ec03163bb2030c5d61e3"}
Sep 11 14:20:50.618182 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:50.618128    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br" podStartSLOduration=1.618112113 podStartE2EDuration="1.618112113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:50.616674985 +0000 UTC m=+1384.580289970" watchObservedRunningTime="2026-09-11 14:20:50.618112113 +0000 UTC m=+1384.581727087"
Sep 11 14:20:53.354634 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:53.354597    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:55.068343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:55.068309    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:55.996234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:55.996197    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:56.976586 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:56.976540    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:57.245960 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:57.245836    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:58.166909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:58.166877    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:59.010920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:59.010884    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:20:59.912678 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:20:59.912633    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:21:00.169476 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:00.169389    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:21:00.169476 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:00.169443    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:21:00.182156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:00.182126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:21:00.634596 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:00.634569    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:21:03.774446 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:03.774414    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:21:05.738918 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:05.738884    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:21:25.292952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.292918    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"]
Sep 11 14:21:25.296530 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.296511    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.299759 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.299736    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-557930d3\"/\"kube-root-ca.crt\""
Sep 11 14:21:25.301207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.301183    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-557930d3\"/\"default-dockercfg-76mn7\""
Sep 11 14:21:25.301207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.301201    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-557930d3\"/\"openshift-service-ca.crt\""
Sep 11 14:21:25.301358 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.301206    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-557930d3\"/\"e2e-lifecycle-model-557930d3-kserve-self-signed-certs\""
Sep 11 14:21:25.310745 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.310721    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"]
Sep 11 14:21:25.319739 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.319716    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:21:25.395283 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.395244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmflh\" (UniqueName: \"kubernetes.io/projected/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-kube-api-access-mmflh\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.395283 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.395281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-dshm\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.395536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.395340    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-model-cache\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.395536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.395355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tmp-dir\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.395536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.395383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tls-certs\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.395536 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.395450    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-home\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.496835 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.496802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmflh\" (UniqueName: \"kubernetes.io/projected/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-kube-api-access-mmflh\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.496835 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.496841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-dshm\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.496895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-model-cache\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.496917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tmp-dir\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.496956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tls-certs\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.496992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-home\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497419 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.497397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-home\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497476 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.497420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tmp-dir\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.497512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.497480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-model-cache\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.499253 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.499233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-dshm\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.499499 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.499480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tls-certs\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.506786 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.506762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmflh\" (UniqueName: \"kubernetes.io/projected/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-kube-api-access-mmflh\") pod \"e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") " pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.605864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.605778    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:25.885397 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:25.885206    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"]
Sep 11 14:21:25.888106 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:21:25.888068    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1150e161_c2e3_49d6_a1b4_9a47cbdaa0f6.slice/crio-d78863b2f5fc25ff4a1fec64075fdca4f7d06128bfaf952cbc6c3ed5aaa3e841 WatchSource:0}: Error finding container d78863b2f5fc25ff4a1fec64075fdca4f7d06128bfaf952cbc6c3ed5aaa3e841: Status 404 returned error can't find the container with id d78863b2f5fc25ff4a1fec64075fdca4f7d06128bfaf952cbc6c3ed5aaa3e841
Sep 11 14:21:26.750625 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:26.750582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx" event={"ID":"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6","Type":"ContainerStarted","Data":"eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401"}
Sep 11 14:21:26.751009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:26.750634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx" event={"ID":"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6","Type":"ContainerStarted","Data":"d78863b2f5fc25ff4a1fec64075fdca4f7d06128bfaf952cbc6c3ed5aaa3e841"}
Sep 11 14:21:26.777118 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:26.777060    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx" podStartSLOduration=1.777040167 podStartE2EDuration="1.777040167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:26.775174858 +0000 UTC m=+1420.738789831" watchObservedRunningTime="2026-09-11 14:21:26.777040167 +0000 UTC m=+1420.740655132"
Sep 11 14:21:30.197260 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:30.197222    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:21:35.605996 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:35.605958    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:35.606496 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:35.606009    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:35.619075 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:35.619051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:35.805548 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:35.805519    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:21:46.371014 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:46.370930    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:21:46.637003 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:46.636928    2558 scope.go:117] "RemoveContainer" containerID="99ef5c05596073bcfc8f2fa58b95b1a58851e8092272fb4362bf48582cdcffce"
Sep 11 14:21:55.169210 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:55.169174    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:21:59.983338 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:21:59.983310    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:01.127157 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:01.127113    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:02.028599 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.028566    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:02.716975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.716946    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"]
Sep 11 14:22:02.721718 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.721696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.725241 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.725214    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"default-dockercfg-fx44k\""
Sep 11 14:22:02.725414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.725394    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"kube-root-ca.crt\""
Sep 11 14:22:02.725524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.725401    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"auth-test-model-031a56-kserve-self-signed-certs\""
Sep 11 14:22:02.725590 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.725394    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"openshift-service-ca.crt\""
Sep 11 14:22:02.739493 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.739471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"]
Sep 11 14:22:02.781101 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.781069    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:02.829333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.829302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-home\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.829481 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.829352    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zqp\" (UniqueName: \"kubernetes.io/projected/c66d8471-9127-465a-b3b6-9f0a43b1eec8-kube-api-access-25zqp\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.829481 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.829432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-model-cache\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.829576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.829491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tls-certs\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.829576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.829519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-dshm\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.829576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.829541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tmp-dir\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.930728 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.930692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zqp\" (UniqueName: \"kubernetes.io/projected/c66d8471-9127-465a-b3b6-9f0a43b1eec8-kube-api-access-25zqp\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.930901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.930738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-model-cache\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.930901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.930785    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tls-certs\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.930901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.930811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-dshm\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.930901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.930830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tmp-dir\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.930901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.930860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-home\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.931272 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.931239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-model-cache\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.931423 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.931351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-home\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.931477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.931443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tmp-dir\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.933200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.933178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-dshm\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.933484 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.933466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tls-certs\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:02.940733 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:02.940709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zqp\" (UniqueName: \"kubernetes.io/projected/c66d8471-9127-465a-b3b6-9f0a43b1eec8-kube-api-access-25zqp\") pod \"auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") " pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:03.031975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.031892    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:03.088350 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.088257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:03.310329 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.310251    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"]
Sep 11 14:22:03.313365 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:22:03.313342    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc66d8471_9127_465a_b3b6_9f0a43b1eec8.slice/crio-254e3e3a2d44dc326dd1bfb3aafdc6f9c747b8fb4aaa4624c99c09dc264987a4 WatchSource:0}: Error finding container 254e3e3a2d44dc326dd1bfb3aafdc6f9c747b8fb4aaa4624c99c09dc264987a4: Status 404 returned error can't find the container with id 254e3e3a2d44dc326dd1bfb3aafdc6f9c747b8fb4aaa4624c99c09dc264987a4
Sep 11 14:22:03.903984 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.903939    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:03.931525 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.931488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" event={"ID":"c66d8471-9127-465a-b3b6-9f0a43b1eec8","Type":"ContainerStarted","Data":"82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87"}
Sep 11 14:22:03.931677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.931536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" event={"ID":"c66d8471-9127-465a-b3b6-9f0a43b1eec8","Type":"ContainerStarted","Data":"254e3e3a2d44dc326dd1bfb3aafdc6f9c747b8fb4aaa4624c99c09dc264987a4"}
Sep 11 14:22:03.960951 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:03.960894    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" podStartSLOduration=1.960880237 podStartE2EDuration="1.960880237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:03.957851756 +0000 UTC m=+1457.921466730" watchObservedRunningTime="2026-09-11 14:22:03.960880237 +0000 UTC m=+1457.924495275"
Sep 11 14:22:04.893063 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:04.893016    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:11.786393 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:11.786356    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:22:13.033097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:13.033059    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:13.033493 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:13.033114    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:13.045143 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:13.045122    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:13.989037 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:13.989005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:22:14.615186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.615149    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"]
Sep 11 14:22:14.615580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.615414    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" podUID="d5beed71-bb68-482f-b7f8-b3cb4fee462b" containerName="maas-api" containerID="cri-o://ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5" gracePeriod=30
Sep 11 14:22:14.641090 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.641060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:14.688608 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.688575    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"]
Sep 11 14:22:14.688903 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.688822    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4" podUID="bd36df79-c545-46ef-a01d-bdeaf1d228aa" containerName="payload-processing" containerID="cri-o://85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc" gracePeriod=30
Sep 11 14:22:14.751542 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.751508    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:14.855057 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.854965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:14.930066 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.930040    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:22:14.956659 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.956635    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:22:14.963482 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.963451    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:14.984540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.984496    2558 generic.go:356] "Generic (PLEG): container finished" podID="d5beed71-bb68-482f-b7f8-b3cb4fee462b" containerID="ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5" exitCode=0
Sep 11 14:22:14.984724 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.984567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" event={"ID":"d5beed71-bb68-482f-b7f8-b3cb4fee462b","Type":"ContainerDied","Data":"ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5"}
Sep 11 14:22:14.984724 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.984599    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"
Sep 11 14:22:14.984724 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.984655    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr" event={"ID":"d5beed71-bb68-482f-b7f8-b3cb4fee462b","Type":"ContainerDied","Data":"9410940ec5078e2066791fe98b3188e712bd8b4d7246ea43985a6509dbba9ec8"}
Sep 11 14:22:14.984724 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.984713    2558 scope.go:117] "RemoveContainer" containerID="ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5"
Sep 11 14:22:14.990477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.989958    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd36df79-c545-46ef-a01d-bdeaf1d228aa" containerID="85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc" exitCode=0
Sep 11 14:22:14.990690 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.990668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4" event={"ID":"bd36df79-c545-46ef-a01d-bdeaf1d228aa","Type":"ContainerDied","Data":"85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc"}
Sep 11 14:22:14.990805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.990702    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"
Sep 11 14:22:14.990805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:14.990717    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4" event={"ID":"bd36df79-c545-46ef-a01d-bdeaf1d228aa","Type":"ContainerDied","Data":"3e7aecbea2878f3535e85c971254eaa728580fbb67ca7a7988a3c4dda89109fc"}
Sep 11 14:22:15.001079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.001054    2558 scope.go:117] "RemoveContainer" containerID="ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5"
Sep 11 14:22:15.001467 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:22:15.001430    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5\": container with ID starting with ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5 not found: ID does not exist" containerID="ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5"
Sep 11 14:22:15.001587 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.001469    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5"} err="failed to get container status \"ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5\": rpc error: code = NotFound desc = could not find container \"ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5\": container with ID starting with ffb57263f8fd24cbfb0fec0979569181f170f398eab117eff02a547bcec52eb5 not found: ID does not exist"
Sep 11 14:22:15.001587 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.001494    2558 scope.go:117] "RemoveContainer" containerID="85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc"
Sep 11 14:22:15.010257 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.010238    2558 scope.go:117] "RemoveContainer" containerID="85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc"
Sep 11 14:22:15.010539 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:22:15.010521    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc\": container with ID starting with 85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc not found: ID does not exist" containerID="85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc"
Sep 11 14:22:15.010605 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.010545    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc"} err="failed to get container status \"85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc\": rpc error: code = NotFound desc = could not find container \"85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc\": container with ID starting with 85d3b89d42867f11f07c50814f85b5a87f6d64b50a68b6aada0bc233a33d61cc not found: ID does not exist"
Sep 11 14:22:15.047911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.047866    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls\") pod \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") "
Sep 11 14:22:15.048104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.047933    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8srdc\" (UniqueName: \"kubernetes.io/projected/bd36df79-c545-46ef-a01d-bdeaf1d228aa-kube-api-access-8srdc\") pod \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") "
Sep 11 14:22:15.048104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.047975    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-metrics-certs\") pod \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") "
Sep 11 14:22:15.048104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.048011    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5beed71-bb68-482f-b7f8-b3cb4fee462b-openshift-service-ca\") pod \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") "
Sep 11 14:22:15.048104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.048100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36df79-c545-46ef-a01d-bdeaf1d228aa-plugins-config-volume\") pod \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\" (UID: \"bd36df79-c545-46ef-a01d-bdeaf1d228aa\") "
Sep 11 14:22:15.048322 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.048128    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcch9\" (UniqueName: \"kubernetes.io/projected/d5beed71-bb68-482f-b7f8-b3cb4fee462b-kube-api-access-pcch9\") pod \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\" (UID: \"d5beed71-bb68-482f-b7f8-b3cb4fee462b\") "
Sep 11 14:22:15.048531 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.048486    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5beed71-bb68-482f-b7f8-b3cb4fee462b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5beed71-bb68-482f-b7f8-b3cb4fee462b" (UID: "d5beed71-bb68-482f-b7f8-b3cb4fee462b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:15.048531 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.048494    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd36df79-c545-46ef-a01d-bdeaf1d228aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd36df79-c545-46ef-a01d-bdeaf1d228aa" (UID: "bd36df79-c545-46ef-a01d-bdeaf1d228aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:15.050220 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.050192    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5beed71-bb68-482f-b7f8-b3cb4fee462b" (UID: "d5beed71-bb68-482f-b7f8-b3cb4fee462b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:22:15.050338 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.050311    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5beed71-bb68-482f-b7f8-b3cb4fee462b-kube-api-access-pcch9" (OuterVolumeSpecName: "kube-api-access-pcch9") pod "d5beed71-bb68-482f-b7f8-b3cb4fee462b" (UID: "d5beed71-bb68-482f-b7f8-b3cb4fee462b"). InnerVolumeSpecName "kube-api-access-pcch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:22:15.050509 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.050488    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5beed71-bb68-482f-b7f8-b3cb4fee462b" (UID: "d5beed71-bb68-482f-b7f8-b3cb4fee462b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:22:15.050596 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.050489    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd36df79-c545-46ef-a01d-bdeaf1d228aa-kube-api-access-8srdc" (OuterVolumeSpecName: "kube-api-access-8srdc") pod "bd36df79-c545-46ef-a01d-bdeaf1d228aa" (UID: "bd36df79-c545-46ef-a01d-bdeaf1d228aa"). InnerVolumeSpecName "kube-api-access-8srdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:22:15.149346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.149253    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srdc\" (UniqueName: \"kubernetes.io/projected/bd36df79-c545-46ef-a01d-bdeaf1d228aa-kube-api-access-8srdc\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:15.149346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.149282    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:15.149346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.149294    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5beed71-bb68-482f-b7f8-b3cb4fee462b-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:15.149346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.149302    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36df79-c545-46ef-a01d-bdeaf1d228aa-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:15.149346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.149311    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcch9\" (UniqueName: \"kubernetes.io/projected/d5beed71-bb68-482f-b7f8-b3cb4fee462b-kube-api-access-pcch9\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:15.149621 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.149390    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5beed71-bb68-482f-b7f8-b3cb4fee462b-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:15.213082 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.213049    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:15.317825 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.317792    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"]
Sep 11 14:22:15.324728 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.324704    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557930d3-5b594dc7c9-lgnmr"]
Sep 11 14:22:15.342655 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.342625    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"]
Sep 11 14:22:15.353512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.353489    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557930d3-5c5bbbf645-knpr4"]
Sep 11 14:22:15.506726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:15.506694    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:16.365437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:16.365405    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:16.504765 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:16.504734    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd36df79-c545-46ef-a01d-bdeaf1d228aa" path="/var/lib/kubelet/pods/bd36df79-c545-46ef-a01d-bdeaf1d228aa/volumes"
Sep 11 14:22:16.505134 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:16.505120    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5beed71-bb68-482f-b7f8-b3cb4fee462b" path="/var/lib/kubelet/pods/d5beed71-bb68-482f-b7f8-b3cb4fee462b/volumes"
Sep 11 14:22:18.187774 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:18.187737    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:18.691695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:18.691655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:19.118638 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:19.118551    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:20.048695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:20.048660    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:23.662628 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:23.662594    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"]
Sep 11 14:22:23.759864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:23.759833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:22:25.873270 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:25.873230    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:22:38.681542 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.681475    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx" podUID="1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" containerName="main" containerID="cri-o://eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401" gracePeriod=45
Sep 11 14:22:38.925258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.925233    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:22:38.966585 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966550    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tls-certs\") pod \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") "
Sep 11 14:22:38.966777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966621    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-model-cache\") pod \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") "
Sep 11 14:22:38.966777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966693    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-dshm\") pod \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") "
Sep 11 14:22:38.966777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966729    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmflh\" (UniqueName: \"kubernetes.io/projected/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-kube-api-access-mmflh\") pod \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") "
Sep 11 14:22:38.966777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966759    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tmp-dir\") pod \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") "
Sep 11 14:22:38.966997 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966853    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-home\") pod \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\" (UID: \"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6\") "
Sep 11 14:22:38.966997 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.966930    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" (UID: "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:38.967152 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.967123    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" (UID: "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:38.967251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.967215    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-home" (OuterVolumeSpecName: "home") pod "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" (UID: "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:38.967251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.967220    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:38.969080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.969046    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-kube-api-access-mmflh" (OuterVolumeSpecName: "kube-api-access-mmflh") pod "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" (UID: "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6"). InnerVolumeSpecName "kube-api-access-mmflh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:22:38.969080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.969072    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" (UID: "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:22:38.969229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:38.969213    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-dshm" (OuterVolumeSpecName: "dshm") pod "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" (UID: "1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:39.068700 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.068662    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:39.068700 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.068695    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:39.068700 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.068704    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:39.068700 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.068711    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:39.068979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.068720    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmflh\" (UniqueName: \"kubernetes.io/projected/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6-kube-api-access-mmflh\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:22:39.102777 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.102743    2558 generic.go:356] "Generic (PLEG): container finished" podID="1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" containerID="eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401" exitCode=0
Sep 11 14:22:39.102907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.102785    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"
Sep 11 14:22:39.102907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.102832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx" event={"ID":"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6","Type":"ContainerDied","Data":"eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401"}
Sep 11 14:22:39.102907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.102877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx" event={"ID":"1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6","Type":"ContainerDied","Data":"d78863b2f5fc25ff4a1fec64075fdca4f7d06128bfaf952cbc6c3ed5aaa3e841"}
Sep 11 14:22:39.102907 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.102900    2558 scope.go:117] "RemoveContainer" containerID="eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401"
Sep 11 14:22:39.112320 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.112301    2558 scope.go:117] "RemoveContainer" containerID="eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401"
Sep 11 14:22:39.112578 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:22:39.112560    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401\": container with ID starting with eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401 not found: ID does not exist" containerID="eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401"
Sep 11 14:22:39.112647 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.112585    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401"} err="failed to get container status \"eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401\": rpc error: code = NotFound desc = could not find container \"eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401\": container with ID starting with eb2f43a5536d476aeabf4524f9cbb3edaa4f1a93ab21199f469c0f2729374401 not found: ID does not exist"
Sep 11 14:22:39.144106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.144081    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"]
Sep 11 14:22:39.166619 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:39.166593    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-557930d3/e2e-lifecycle-model-557930d3-kserve-7bc6bd5b4c-bs6qx"]
Sep 11 14:22:40.503629 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:40.503594    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" path="/var/lib/kubelet/pods/1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6/volumes"
Sep 11 14:22:45.567523 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:45.567488    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:22:46.532018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:46.531992    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:22:46.535081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:46.535057    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:22:58.575755 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:22:58.575710    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:23:01.888509 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:01.888468    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:02.082875 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:02.082843    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:09.312895 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:09.312813    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:09.407805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:09.407770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:09.985249 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:09.985216    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:10.082335 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:10.082297    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:10.826073 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:10.826012    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:10.936612 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:10.936577    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:12.400806 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:12.400770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:15.189023 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:15.188989    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:23:19.070915 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:19.070882    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:19.193646 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:19.193612    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:19.775643 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:19.775612    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:26.647729 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:26.647692    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:26.778496 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:26.778464    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:26.894910 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:26.894873    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:31.487802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:31.487761    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:23:37.361479 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:37.361439    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:38.934974 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:38.934934    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:23:46.576245 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:46.576211    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:23:58.575104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:23:58.575061    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:02.090781 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:02.090746    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:24:05.612885 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:05.612842    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:05.710219 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:05.710174    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:05.979219 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:05.979175    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:06.344862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:06.344764    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:06.505589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:06.505556    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:08.997560 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:08.997524    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:09.655729 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:09.655696    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:09.754529 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:09.754479    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:10.153808 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:10.153771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:10.270556 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:10.270521    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:11.491414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:11.491376    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:12.664081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:12.664021    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:13.617939 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:13.617907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:14.519784 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:14.519752    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:14.618792 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:14.618763    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:15.145409 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.145377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:15.256638 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.256604    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:15.330679 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.330646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"]
Sep 11 14:24:15.331023 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331011    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5beed71-bb68-482f-b7f8-b3cb4fee462b" containerName="maas-api"
Sep 11 14:24:15.331023 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331037    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5beed71-bb68-482f-b7f8-b3cb4fee462b" containerName="maas-api"
Sep 11 14:24:15.331139 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331055    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd36df79-c545-46ef-a01d-bdeaf1d228aa" containerName="payload-processing"
Sep 11 14:24:15.331139 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331061    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd36df79-c545-46ef-a01d-bdeaf1d228aa" containerName="payload-processing"
Sep 11 14:24:15.331139 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331072    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" containerName="main"
Sep 11 14:24:15.331139 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331079    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" containerName="main"
Sep 11 14:24:15.331139 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331139    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5beed71-bb68-482f-b7f8-b3cb4fee462b" containerName="maas-api"
Sep 11 14:24:15.331296 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331148    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd36df79-c545-46ef-a01d-bdeaf1d228aa" containerName="payload-processing"
Sep 11 14:24:15.331296 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.331157    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1150e161-c2e3-49d6-a1b4-9a47cbdaa0f6" containerName="main"
Sep 11 14:24:15.334344 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.334326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.338682 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.338659    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d52edb-dockercfg-cghzm\""
Sep 11 14:24:15.338942 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.338927    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d52edb\""
Sep 11 14:24:15.351229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.351197    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"]
Sep 11 14:24:15.475106 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.475055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151fdb49-9ea5-4787-ae20-80ed9405f360-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.475303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.475179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksts4\" (UniqueName: \"kubernetes.io/projected/151fdb49-9ea5-4787-ae20-80ed9405f360-kube-api-access-ksts4\") pod \"payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.579101 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.579062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151fdb49-9ea5-4787-ae20-80ed9405f360-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.579590 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.579182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksts4\" (UniqueName: \"kubernetes.io/projected/151fdb49-9ea5-4787-ae20-80ed9405f360-kube-api-access-ksts4\") pod \"payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.579823 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.579800    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151fdb49-9ea5-4787-ae20-80ed9405f360-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.589450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.589416    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksts4\" (UniqueName: \"kubernetes.io/projected/151fdb49-9ea5-4787-ae20-80ed9405f360-kube-api-access-ksts4\") pod \"payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.644053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.643992    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:15.919991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:15.919881    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"]
Sep 11 14:24:15.922701 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:24:15.922672    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151fdb49_9ea5_4787_ae20_80ed9405f360.slice/crio-c25b9cdaacbbadeaf98c47b4f7a03a56f388593bf816ee382fbbcd7d43700b89 WatchSource:0}: Error finding container c25b9cdaacbbadeaf98c47b4f7a03a56f388593bf816ee382fbbcd7d43700b89: Status 404 returned error can't find the container with id c25b9cdaacbbadeaf98c47b4f7a03a56f388593bf816ee382fbbcd7d43700b89
Sep 11 14:24:16.579387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:16.579341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27" event={"ID":"151fdb49-9ea5-4787-ae20-80ed9405f360","Type":"ContainerStarted","Data":"f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c"}
Sep 11 14:24:16.579789 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:16.579403    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27" event={"ID":"151fdb49-9ea5-4787-ae20-80ed9405f360","Type":"ContainerStarted","Data":"c25b9cdaacbbadeaf98c47b4f7a03a56f388593bf816ee382fbbcd7d43700b89"}
Sep 11 14:24:16.579789 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:16.579428    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:16.600453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:16.600404    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27" podStartSLOduration=1.600388751 podStartE2EDuration="1.600388751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:16.598258043 +0000 UTC m=+1590.561873058" watchObservedRunningTime="2026-09-11 14:24:16.600388751 +0000 UTC m=+1590.564003741"
Sep 11 14:24:18.524395 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:18.524355    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:18.862869 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:18.862784    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:24:27.025017 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:27.024976    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:27.586586 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:27.586553    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:24:27.885051 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:27.884957    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"]
Sep 11 14:24:27.924973 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:27.924932    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:28.964318 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:28.964286    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"]
Sep 11 14:24:28.994626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:28.994579    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:37.169235 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:37.169184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:37.973468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:37.973432    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:38.843346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:38.843312    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:39.829135 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:39.829099    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:39.930821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:39.930786    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:40.337540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:40.337506    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:40.727758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:40.727726    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:41.084752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:41.084664    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:41.211597 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:41.211565    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:42.202081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:42.202046    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:42.903913 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:42.903864    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br" podUID="46254fe0-1e45-43d0-aa60-23ddc24d372f" containerName="main" containerID="cri-o://2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c" gracePeriod=45
Sep 11 14:24:43.150406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.150377    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:24:43.245218 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245183    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlljw\" (UniqueName: \"kubernetes.io/projected/46254fe0-1e45-43d0-aa60-23ddc24d372f-kube-api-access-mlljw\") pod \"46254fe0-1e45-43d0-aa60-23ddc24d372f\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") "
Sep 11 14:24:43.245656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245345    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fe0-1e45-43d0-aa60-23ddc24d372f-tls-certs\") pod \"46254fe0-1e45-43d0-aa60-23ddc24d372f\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") "
Sep 11 14:24:43.245656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245378    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-model-cache\") pod \"46254fe0-1e45-43d0-aa60-23ddc24d372f\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") "
Sep 11 14:24:43.245656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245396    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-dshm\") pod \"46254fe0-1e45-43d0-aa60-23ddc24d372f\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") "
Sep 11 14:24:43.245656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245470    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-home\") pod \"46254fe0-1e45-43d0-aa60-23ddc24d372f\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") "
Sep 11 14:24:43.245656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245499    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-tmp-dir\") pod \"46254fe0-1e45-43d0-aa60-23ddc24d372f\" (UID: \"46254fe0-1e45-43d0-aa60-23ddc24d372f\") "
Sep 11 14:24:43.245656 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-model-cache" (OuterVolumeSpecName: "model-cache") pod "46254fe0-1e45-43d0-aa60-23ddc24d372f" (UID: "46254fe0-1e45-43d0-aa60-23ddc24d372f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:43.246093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245695    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-home" (OuterVolumeSpecName: "home") pod "46254fe0-1e45-43d0-aa60-23ddc24d372f" (UID: "46254fe0-1e45-43d0-aa60-23ddc24d372f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:43.246093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46254fe0-1e45-43d0-aa60-23ddc24d372f" (UID: "46254fe0-1e45-43d0-aa60-23ddc24d372f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:43.246093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245900    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:43.246093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245913    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:43.246093 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.245922    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:43.247661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.247629    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-dshm" (OuterVolumeSpecName: "dshm") pod "46254fe0-1e45-43d0-aa60-23ddc24d372f" (UID: "46254fe0-1e45-43d0-aa60-23ddc24d372f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:43.247661 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.247644    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46254fe0-1e45-43d0-aa60-23ddc24d372f-kube-api-access-mlljw" (OuterVolumeSpecName: "kube-api-access-mlljw") pod "46254fe0-1e45-43d0-aa60-23ddc24d372f" (UID: "46254fe0-1e45-43d0-aa60-23ddc24d372f"). InnerVolumeSpecName "kube-api-access-mlljw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:24:43.247824 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.247690    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46254fe0-1e45-43d0-aa60-23ddc24d372f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46254fe0-1e45-43d0-aa60-23ddc24d372f" (UID: "46254fe0-1e45-43d0-aa60-23ddc24d372f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:24:43.347091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.347017    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlljw\" (UniqueName: \"kubernetes.io/projected/46254fe0-1e45-43d0-aa60-23ddc24d372f-kube-api-access-mlljw\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:43.347091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.347086    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fe0-1e45-43d0-aa60-23ddc24d372f-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:43.347091 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.347096    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fe0-1e45-43d0-aa60-23ddc24d372f-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:43.491434 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.491395    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:43.711074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.711016    2558 generic.go:356] "Generic (PLEG): container finished" podID="46254fe0-1e45-43d0-aa60-23ddc24d372f" containerID="2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c" exitCode=0
Sep 11 14:24:43.711074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.711064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br" event={"ID":"46254fe0-1e45-43d0-aa60-23ddc24d372f","Type":"ContainerDied","Data":"2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c"}
Sep 11 14:24:43.711343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.711093    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"
Sep 11 14:24:43.711343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.711107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br" event={"ID":"46254fe0-1e45-43d0-aa60-23ddc24d372f","Type":"ContainerDied","Data":"9d65d4a5e48e64ebe9800a3061d976763de874417929ec03163bb2030c5d61e3"}
Sep 11 14:24:43.711343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.711125    2558 scope.go:117] "RemoveContainer" containerID="2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c"
Sep 11 14:24:43.721073 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.721045    2558 scope.go:117] "RemoveContainer" containerID="2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c"
Sep 11 14:24:43.721390 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:24:43.721367    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c\": container with ID starting with 2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c not found: ID does not exist" containerID="2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c"
Sep 11 14:24:43.721456 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.721401    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c"} err="failed to get container status \"2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c\": rpc error: code = NotFound desc = could not find container \"2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c\": container with ID starting with 2c6027190912d25ff95eb425d94931da9d13fc4b817d542e7a6fc41e0cf75d0c not found: ID does not exist"
Sep 11 14:24:43.738364 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.738332    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"]
Sep 11 14:24:43.740376 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.740347    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/auth-test-model-1e2e7d-kserve-7fdcc458cf-8w8br"]
Sep 11 14:24:43.983228 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.983135    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" podUID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" containerName="main" containerID="cri-o://82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87" gracePeriod=45
Sep 11 14:24:43.987074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:43.987042    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" podUID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": EOF"
Sep 11 14:24:44.001490 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.001469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:44.088215 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.088182    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:44.229669 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.229643    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:24:44.298471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.298385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:24:44.358207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tmp-dir\") pod \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") "
Sep 11 14:24:44.358399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358236    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-dshm\") pod \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") "
Sep 11 14:24:44.358399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-home\") pod \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") "
Sep 11 14:24:44.358399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358326    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-model-cache\") pod \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") "
Sep 11 14:24:44.358399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358370    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tls-certs\") pod \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") "
Sep 11 14:24:44.358598 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358416    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25zqp\" (UniqueName: \"kubernetes.io/projected/c66d8471-9127-465a-b3b6-9f0a43b1eec8-kube-api-access-25zqp\") pod \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\" (UID: \"c66d8471-9127-465a-b3b6-9f0a43b1eec8\") "
Sep 11 14:24:44.358598 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358487    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c66d8471-9127-465a-b3b6-9f0a43b1eec8" (UID: "c66d8471-9127-465a-b3b6-9f0a43b1eec8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:44.358598 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358501    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-home" (OuterVolumeSpecName: "home") pod "c66d8471-9127-465a-b3b6-9f0a43b1eec8" (UID: "c66d8471-9127-465a-b3b6-9f0a43b1eec8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:44.358721 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c66d8471-9127-465a-b3b6-9f0a43b1eec8" (UID: "c66d8471-9127-465a-b3b6-9f0a43b1eec8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:44.358854 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.358831    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:44.359060 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.359018    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:44.359060 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.359061    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:44.360912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.360883    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66d8471-9127-465a-b3b6-9f0a43b1eec8-kube-api-access-25zqp" (OuterVolumeSpecName: "kube-api-access-25zqp") pod "c66d8471-9127-465a-b3b6-9f0a43b1eec8" (UID: "c66d8471-9127-465a-b3b6-9f0a43b1eec8"). InnerVolumeSpecName "kube-api-access-25zqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:24:44.360990 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.360902    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c66d8471-9127-465a-b3b6-9f0a43b1eec8" (UID: "c66d8471-9127-465a-b3b6-9f0a43b1eec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:24:44.360990 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.360886    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-dshm" (OuterVolumeSpecName: "dshm") pod "c66d8471-9127-465a-b3b6-9f0a43b1eec8" (UID: "c66d8471-9127-465a-b3b6-9f0a43b1eec8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:24:44.460485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.460448    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66d8471-9127-465a-b3b6-9f0a43b1eec8-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:44.460485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.460479    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66d8471-9127-465a-b3b6-9f0a43b1eec8-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:44.460485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.460491    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25zqp\" (UniqueName: \"kubernetes.io/projected/c66d8471-9127-465a-b3b6-9f0a43b1eec8-kube-api-access-25zqp\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:24:44.488383 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.488351    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:44.503948 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.503912    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46254fe0-1e45-43d0-aa60-23ddc24d372f" path="/var/lib/kubelet/pods/46254fe0-1e45-43d0-aa60-23ddc24d372f/volumes"
Sep 11 14:24:44.717805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.717774    2558 generic.go:356] "Generic (PLEG): container finished" podID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" containerID="82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87" exitCode=0
Sep 11 14:24:44.718000 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.717817    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"
Sep 11 14:24:44.718000 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.717826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" event={"ID":"c66d8471-9127-465a-b3b6-9f0a43b1eec8","Type":"ContainerDied","Data":"82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87"}
Sep 11 14:24:44.718000 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.717859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9" event={"ID":"c66d8471-9127-465a-b3b6-9f0a43b1eec8","Type":"ContainerDied","Data":"254e3e3a2d44dc326dd1bfb3aafdc6f9c747b8fb4aaa4624c99c09dc264987a4"}
Sep 11 14:24:44.718000 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.717885    2558 scope.go:117] "RemoveContainer" containerID="82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87"
Sep 11 14:24:44.726435 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.726415    2558 scope.go:117] "RemoveContainer" containerID="82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87"
Sep 11 14:24:44.726714 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:24:44.726699    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87\": container with ID starting with 82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87 not found: ID does not exist" containerID="82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87"
Sep 11 14:24:44.726761 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.726721    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87"} err="failed to get container status \"82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87\": rpc error: code = NotFound desc = could not find container \"82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87\": container with ID starting with 82a38a76fb98cf2f94ce9afb0d24628d9695275f4538b0cc18278caed750ed87 not found: ID does not exist"
Sep 11 14:24:44.740671 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.740645    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"]
Sep 11 14:24:44.746272 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:44.746248    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/auth-test-model-031a56-kserve-6f4c8ffddc-pt8j9"]
Sep 11 14:24:45.615200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:45.615165    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:46.457394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:46.457360    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:46.504015 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:46.503982    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" path="/var/lib/kubelet/pods/c66d8471-9127-465a-b3b6-9f0a43b1eec8/volumes"
Sep 11 14:24:47.319748 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:47.319715    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:48.275521 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:48.275484    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:49.078630 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.078597    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"]
Sep 11 14:24:49.079161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.079143    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46254fe0-1e45-43d0-aa60-23ddc24d372f" containerName="main"
Sep 11 14:24:49.079258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.079164    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="46254fe0-1e45-43d0-aa60-23ddc24d372f" containerName="main"
Sep 11 14:24:49.079258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.079189    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" containerName="main"
Sep 11 14:24:49.079258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.079198    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" containerName="main"
Sep 11 14:24:49.079411 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.079280    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="46254fe0-1e45-43d0-aa60-23ddc24d372f" containerName="main"
Sep 11 14:24:49.079411 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.079297    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66d8471-9127-465a-b3b6-9f0a43b1eec8" containerName="main"
Sep 11 14:24:49.082221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.082199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.085189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.085169    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:24:49.086066 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.086007    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:24:49.086066 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.086051    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:24:49.086282 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.086049    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bfa49a-serving-cert\""
Sep 11 14:24:49.087472 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.087456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktm9c\""
Sep 11 14:24:49.099874 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.099849    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"]
Sep 11 14:24:49.134509 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.134473    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:49.204418 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.204373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bjj\" (UniqueName: \"kubernetes.io/projected/76af003b-80b2-4d8c-8914-3e9b2c307a0c-kube-api-access-j9bjj\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.204626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.204433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76af003b-80b2-4d8c-8914-3e9b2c307a0c-openshift-service-ca\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.204626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.204550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-maas-api-tls\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.204626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.204611    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-metrics-certs\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.236588 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.236557    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:49.305254 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.305216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-maas-api-tls\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.305453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.305277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-metrics-certs\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.305453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.305323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bjj\" (UniqueName: \"kubernetes.io/projected/76af003b-80b2-4d8c-8914-3e9b2c307a0c-kube-api-access-j9bjj\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.305453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.305377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76af003b-80b2-4d8c-8914-3e9b2c307a0c-openshift-service-ca\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.306081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.306055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76af003b-80b2-4d8c-8914-3e9b2c307a0c-openshift-service-ca\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.307817 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.307787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-metrics-certs\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.307900 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.307829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-maas-api-tls\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.314125 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.314103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bjj\" (UniqueName: \"kubernetes.io/projected/76af003b-80b2-4d8c-8914-3e9b2c307a0c-kube-api-access-j9bjj\") pod \"maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.357159 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.357069    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:49.392999 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.392963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:49.690759 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.690729    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"]
Sep 11 14:24:49.693969 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:24:49.693935    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76af003b_80b2_4d8c_8914_3e9b2c307a0c.slice/crio-a79821b0a81f72df550f14ae78c9aa7e399b3890087c10f3ff771607eba75920 WatchSource:0}: Error finding container a79821b0a81f72df550f14ae78c9aa7e399b3890087c10f3ff771607eba75920: Status 404 returned error can't find the container with id a79821b0a81f72df550f14ae78c9aa7e399b3890087c10f3ff771607eba75920
Sep 11 14:24:49.695552 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.695531    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:24:49.744263 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.744224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs" event={"ID":"76af003b-80b2-4d8c-8914-3e9b2c307a0c","Type":"ContainerStarted","Data":"a79821b0a81f72df550f14ae78c9aa7e399b3890087c10f3ff771607eba75920"}
Sep 11 14:24:49.826043 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.825989    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:49.923970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.923930    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:24:49.957570 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.957467    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"]
Sep 11 14:24:49.962144 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.962117    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:49.964753 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.964728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bfa49a\""
Sep 11 14:24:49.973966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:49.973937    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"]
Sep 11 14:24:50.019457 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.019426    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"]
Sep 11 14:24:50.024036 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.024002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.026690 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.026668    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bfa49a-dockercfg-gjnmh\""
Sep 11 14:24:50.060849 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.060814    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"]
Sep 11 14:24:50.112869 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.112834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4stv\" (UniqueName: \"kubernetes.io/projected/1baf3c75-83f2-47bb-a303-5372130fdb06-kube-api-access-m4stv\") pod \"payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.113294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.112875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44396958-0b5f-4721-a102-0d8a9baa3063-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.113294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.112903    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5dx\" (UniqueName: \"kubernetes.io/projected/44396958-0b5f-4721-a102-0d8a9baa3063-kube-api-access-qv5dx\") pod \"payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.113294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.113022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baf3c75-83f2-47bb-a303-5372130fdb06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.214207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.214118    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baf3c75-83f2-47bb-a303-5372130fdb06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.214207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.214183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4stv\" (UniqueName: \"kubernetes.io/projected/1baf3c75-83f2-47bb-a303-5372130fdb06-kube-api-access-m4stv\") pod \"payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.214450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.214340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44396958-0b5f-4721-a102-0d8a9baa3063-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.214450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.214384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5dx\" (UniqueName: \"kubernetes.io/projected/44396958-0b5f-4721-a102-0d8a9baa3063-kube-api-access-qv5dx\") pod \"payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.214959 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.214935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baf3c75-83f2-47bb-a303-5372130fdb06-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.215076 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.215022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44396958-0b5f-4721-a102-0d8a9baa3063-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.223507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.223484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5dx\" (UniqueName: \"kubernetes.io/projected/44396958-0b5f-4721-a102-0d8a9baa3063-kube-api-access-qv5dx\") pod \"payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.224644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.224616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4stv\" (UniqueName: \"kubernetes.io/projected/1baf3c75-83f2-47bb-a303-5372130fdb06-kube-api-access-m4stv\") pod \"payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") " pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.273614 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.273569    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.338403 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.338370    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.553756 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.553725    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"]
Sep 11 14:24:50.554516 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:24:50.554489    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1baf3c75_83f2_47bb_a303_5372130fdb06.slice/crio-8c75995478f9ad991ec98dc3358da468a58beeedd72e71778962bcbb3e432644 WatchSource:0}: Error finding container 8c75995478f9ad991ec98dc3358da468a58beeedd72e71778962bcbb3e432644: Status 404 returned error can't find the container with id 8c75995478f9ad991ec98dc3358da468a58beeedd72e71778962bcbb3e432644
Sep 11 14:24:50.624887 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.624858    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"]
Sep 11 14:24:50.626508 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:24:50.626482    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44396958_0b5f_4721_a102_0d8a9baa3063.slice/crio-f9209965b1a6a7d58efe39a3dbeb60da893cc6448d85f70e46813e2023e1540b WatchSource:0}: Error finding container f9209965b1a6a7d58efe39a3dbeb60da893cc6448d85f70e46813e2023e1540b: Status 404 returned error can't find the container with id f9209965b1a6a7d58efe39a3dbeb60da893cc6448d85f70e46813e2023e1540b
Sep 11 14:24:50.754121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.754083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g" event={"ID":"44396958-0b5f-4721-a102-0d8a9baa3063","Type":"ContainerStarted","Data":"1e2b69ec1e929df5e8fb26d2a5d3770acbc5ca31fc7f3f7dbb8527c169cbc7b7"}
Sep 11 14:24:50.754387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.754148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g" event={"ID":"44396958-0b5f-4721-a102-0d8a9baa3063","Type":"ContainerStarted","Data":"f9209965b1a6a7d58efe39a3dbeb60da893cc6448d85f70e46813e2023e1540b"}
Sep 11 14:24:50.754387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.754195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:24:50.755729 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.755705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs" event={"ID":"76af003b-80b2-4d8c-8914-3e9b2c307a0c","Type":"ContainerStarted","Data":"3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e"}
Sep 11 14:24:50.755854 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.755755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:50.757309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.757278    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj" event={"ID":"1baf3c75-83f2-47bb-a303-5372130fdb06","Type":"ContainerStarted","Data":"d04b00be8299108abffa21ff3ded02d69cf38a9f4036c45ae8a43af53a0ae2f0"}
Sep 11 14:24:50.757422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.757332    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj" event={"ID":"1baf3c75-83f2-47bb-a303-5372130fdb06","Type":"ContainerStarted","Data":"8c75995478f9ad991ec98dc3358da468a58beeedd72e71778962bcbb3e432644"}
Sep 11 14:24:50.757422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.757375    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:24:50.774133 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.774086    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g" podStartSLOduration=1.774070705 podStartE2EDuration="1.774070705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:50.773882979 +0000 UTC m=+1624.737497947" watchObservedRunningTime="2026-09-11 14:24:50.774070705 +0000 UTC m=+1624.737685679"
Sep 11 14:24:50.797778 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.797723    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs" podStartSLOduration=1.494436027 podStartE2EDuration="1.797707826s" totalImagesPullingTime="303.271799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:50.796979063 +0000 UTC m=+1624.760594050" watchObservedRunningTime="2026-09-11 14:24:50.797707826 +0000 UTC m=+1624.761322827"
Sep 11 14:24:50.819061 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:50.818936    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj" podStartSLOduration=1.8189229519999999 podStartE2EDuration="1.818922952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:50.818134624 +0000 UTC m=+1624.781749600" watchObservedRunningTime="2026-09-11 14:24:50.818922952 +0000 UTC m=+1624.782537925"
Sep 11 14:24:56.769390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:56.769360    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:24:56.919428 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:24:56.919394    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:01.285955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:01.285920    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:01.765061 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:01.765014    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:25:01.765348 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:01.765328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:25:02.883837 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.883794    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"]
Sep 11 14:25:02.887682 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.887647    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:02.892217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.892178    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"default-dockercfg-zvrmr\""
Sep 11 14:25:02.892396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.892220    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"openshift-service-ca.crt\""
Sep 11 14:25:02.892396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.892277    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"sub-test-model-1e2e7d-kserve-self-signed-certs\""
Sep 11 14:25:02.892396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.892349    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"kube-root-ca.crt\""
Sep 11 14:25:02.904080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.904048    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"]
Sep 11 14:25:02.921613 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:02.921575    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:03.033855 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.033818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tmp-dir\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.034088 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.033873    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-dshm\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.034088 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.033899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-home\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.034088 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.033977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tls-certs\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.034088 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.034064    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-model-cache\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.034260 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.034116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-kube-api-access-58mdj\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135059 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.134941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tls-certs\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-model-cache\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-kube-api-access-58mdj\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tmp-dir\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-dshm\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135446 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-home\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135561 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135541    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-model-cache\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135561 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tmp-dir\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.135728 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.135706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-home\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.137411 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.137392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-dshm\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.137598 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.137581    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tls-certs\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.165707 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.165670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-kube-api-access-58mdj\") pod \"sub-test-model-1e2e7d-kserve-668457b7f9-68ftj\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.198726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.198674    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:03.480213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.480188    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"]
Sep 11 14:25:03.482356 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:25:03.482323    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a4e27e4_fdbd_47cd_a4ae_3088c92237ac.slice/crio-53c105525d2543369f41a2aa2324f4e4c954b201f1038d20ef8761b28f78b889 WatchSource:0}: Error finding container 53c105525d2543369f41a2aa2324f4e4c954b201f1038d20ef8761b28f78b889: Status 404 returned error can't find the container with id 53c105525d2543369f41a2aa2324f4e4c954b201f1038d20ef8761b28f78b889
Sep 11 14:25:03.672502 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.672455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:25:03.824740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.824629    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj" event={"ID":"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac","Type":"ContainerStarted","Data":"e90ab570e0498612dd8ffaf21059c5323f96d0894a5d834fb03f3d0e04510331"}
Sep 11 14:25:03.824740 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.824683    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj" event={"ID":"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac","Type":"ContainerStarted","Data":"53c105525d2543369f41a2aa2324f4e4c954b201f1038d20ef8761b28f78b889"}
Sep 11 14:25:03.844227 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:03.844177    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj" podStartSLOduration=1.844160884 podStartE2EDuration="1.844160884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:03.84339203 +0000 UTC m=+1637.807007005" watchObservedRunningTime="2026-09-11 14:25:03.844160884 +0000 UTC m=+1637.807775892"
Sep 11 14:25:04.940884 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:04.940829    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:05.026889 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:05.026849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:10.375335 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:10.375298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:12.077151 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:12.077117    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:13.008261 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:13.008229    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:13.199268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:13.199225    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:13.199697 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:13.199284    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:13.212229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:13.212195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:13.882446 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:13.882411    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:25:14.058480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:14.058444    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:14.171975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:14.171882    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:14.965492 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:14.965456    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:15.934544 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:15.934504    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:16.930305 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:16.930267    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:17.991752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:17.991721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:18.887802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:18.887767    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:19.171165 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:19.171080    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:25:19.810219 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:19.810188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:20.818594 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:20.818553    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:21.709624 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:21.709590    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:22.615023 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:22.614979    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:23.501318 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:23.501285    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:27.414662 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:27.414618    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:29.466909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.466875    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"]
Sep 11 14:25:29.471506 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.471489    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.475150 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.475130    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"sub-test-model-031a56-kserve-self-signed-certs\""
Sep 11 14:25:29.475262 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.475152    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"default-dockercfg-fx44k\""
Sep 11 14:25:29.475489 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.475470    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"kube-root-ca.crt\""
Sep 11 14:25:29.475566 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.475480    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"openshift-service-ca.crt\""
Sep 11 14:25:29.482053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.482012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"]
Sep 11 14:25:29.572823 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.572786    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:29.581930 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.581897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-home\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.582072 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.582008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-model-cache\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.582143 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.582117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gft\" (UniqueName: \"kubernetes.io/projected/e488e71e-8662-4e35-8263-b4e2990b2d39-kube-api-access-46gft\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.582231 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.582216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e488e71e-8662-4e35-8263-b4e2990b2d39-tls-certs\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.582292 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.582252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-tmp-dir\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.582351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.582293    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-dshm\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.682818 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.682783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e488e71e-8662-4e35-8263-b4e2990b2d39-tls-certs\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.682818 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.682823    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-tmp-dir\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683116 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.682846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-dshm\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683116 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.682896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-home\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683116 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.682924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-model-cache\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683116 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.682972    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gft\" (UniqueName: \"kubernetes.io/projected/e488e71e-8662-4e35-8263-b4e2990b2d39-kube-api-access-46gft\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.683313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-tmp-dir\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.683339    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-home\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.683429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.683390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-model-cache\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.685220 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.685189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-dshm\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.685460 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.685444    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e488e71e-8662-4e35-8263-b4e2990b2d39-tls-certs\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.693124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.693095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gft\" (UniqueName: \"kubernetes.io/projected/e488e71e-8662-4e35-8263-b4e2990b2d39-kube-api-access-46gft\") pod \"sub-test-model-031a56-kserve-7464f84cdc-f5q94\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") " pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:29.781911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:29.781809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:30.059816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:30.059792    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"]
Sep 11 14:25:30.061653 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:25:30.061625    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode488e71e_8662_4e35_8263_b4e2990b2d39.slice/crio-41211ff61f363f50ae98b979a099cda9760e505cab87928c59b2ab7d319047a8 WatchSource:0}: Error finding container 41211ff61f363f50ae98b979a099cda9760e505cab87928c59b2ab7d319047a8: Status 404 returned error can't find the container with id 41211ff61f363f50ae98b979a099cda9760e505cab87928c59b2ab7d319047a8
Sep 11 14:25:30.961204 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:30.961168    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94" event={"ID":"e488e71e-8662-4e35-8263-b4e2990b2d39","Type":"ContainerStarted","Data":"30dae9c9b0ffdef5ffee60fa735d76f91f3e5fc7ae813c8725e4c1265b320865"}
Sep 11 14:25:30.961606 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:30.961213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94" event={"ID":"e488e71e-8662-4e35-8263-b4e2990b2d39","Type":"ContainerStarted","Data":"41211ff61f363f50ae98b979a099cda9760e505cab87928c59b2ab7d319047a8"}
Sep 11 14:25:30.984196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:30.984138    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94" podStartSLOduration=1.984121609 podStartE2EDuration="1.984121609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:30.981993953 +0000 UTC m=+1664.945608930" watchObservedRunningTime="2026-09-11 14:25:30.984121609 +0000 UTC m=+1664.947736583"
Sep 11 14:25:38.870665 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:38.870625    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:39.030535 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.030501    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"]
Sep 11 14:25:39.030758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.030711    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27" podUID="151fdb49-9ea5-4787-ae20-80ed9405f360" containerName="payload-pre-processing" containerID="cri-o://f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c" gracePeriod=30
Sep 11 14:25:39.094528 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.094500    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:39.240901 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.240858    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:39.277517 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.277493    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:25:39.370672 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.370641    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151fdb49-9ea5-4787-ae20-80ed9405f360-plugins-config-volume\") pod \"151fdb49-9ea5-4787-ae20-80ed9405f360\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") "
Sep 11 14:25:39.370830 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.370697    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksts4\" (UniqueName: \"kubernetes.io/projected/151fdb49-9ea5-4787-ae20-80ed9405f360-kube-api-access-ksts4\") pod \"151fdb49-9ea5-4787-ae20-80ed9405f360\" (UID: \"151fdb49-9ea5-4787-ae20-80ed9405f360\") "
Sep 11 14:25:39.371133 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.371106    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151fdb49-9ea5-4787-ae20-80ed9405f360-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "151fdb49-9ea5-4787-ae20-80ed9405f360" (UID: "151fdb49-9ea5-4787-ae20-80ed9405f360"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:25:39.372800 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.372775    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151fdb49-9ea5-4787-ae20-80ed9405f360-kube-api-access-ksts4" (OuterVolumeSpecName: "kube-api-access-ksts4") pod "151fdb49-9ea5-4787-ae20-80ed9405f360" (UID: "151fdb49-9ea5-4787-ae20-80ed9405f360"). InnerVolumeSpecName "kube-api-access-ksts4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:25:39.428768 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.428696    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:39.471679 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.471651    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksts4\" (UniqueName: \"kubernetes.io/projected/151fdb49-9ea5-4787-ae20-80ed9405f360-kube-api-access-ksts4\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:25:39.471679 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.471674    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151fdb49-9ea5-4787-ae20-80ed9405f360-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:25:39.556679 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.556637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:39.695746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.695708    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:39.782514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.782486    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:39.782693 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.782532    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:39.795439 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.795407    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:39.863541 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:39.863503    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:40.007288 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.007195    2558 generic.go:356] "Generic (PLEG): container finished" podID="151fdb49-9ea5-4787-ae20-80ed9405f360" containerID="f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c" exitCode=0
Sep 11 14:25:40.007288 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.007239    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"
Sep 11 14:25:40.007797 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.007237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27" event={"ID":"151fdb49-9ea5-4787-ae20-80ed9405f360","Type":"ContainerDied","Data":"f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c"}
Sep 11 14:25:40.007797 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.007363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27" event={"ID":"151fdb49-9ea5-4787-ae20-80ed9405f360","Type":"ContainerDied","Data":"c25b9cdaacbbadeaf98c47b4f7a03a56f388593bf816ee382fbbcd7d43700b89"}
Sep 11 14:25:40.007797 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.007391    2558 scope.go:117] "RemoveContainer" containerID="f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c"
Sep 11 14:25:40.016826 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.016804    2558 scope.go:117] "RemoveContainer" containerID="f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c"
Sep 11 14:25:40.017218 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:25:40.017193    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c\": container with ID starting with f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c not found: ID does not exist" containerID="f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c"
Sep 11 14:25:40.017307 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.017225    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c"} err="failed to get container status \"f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c\": rpc error: code = NotFound desc = could not find container \"f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c\": container with ID starting with f2c714e78348af57337613e6f3e820e53d3297b13d5ab91c508d8883ee99419c not found: ID does not exist"
Sep 11 14:25:40.020692 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.020666    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:25:40.030478 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.030446    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"]
Sep 11 14:25:40.034794 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.034762    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d52edb-7466577fb6-7xn27"]
Sep 11 14:25:40.503839 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.503805    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="151fdb49-9ea5-4787-ae20-80ed9405f360" path="/var/lib/kubelet/pods/151fdb49-9ea5-4787-ae20-80ed9405f360/volumes"
Sep 11 14:25:40.619381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:40.619350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:41.416596 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:41.416550    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:25:41.847293 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:41.847253    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:43.102422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:43.102385    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:44.200139 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:44.200103    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:45.056111 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:45.056082    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:46.146870 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:46.146840    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:47.298575 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:47.298534    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:47.452181 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:47.452142    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:47.655217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:47.655107    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:48.292950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:48.292916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:49.367862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:49.367828    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:50.055388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:50.055335    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:51.033066 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:51.033009    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:51.895664 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:51.895637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:52.693915 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:52.693858    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:25:53.756723 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:25:53.756687    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:02.970761 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:02.970720    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:26:03.749303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.749270    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"]
Sep 11 14:26:03.749503 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.749482    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs" podUID="76af003b-80b2-4d8c-8914-3e9b2c307a0c" containerName="maas-api" containerID="cri-o://3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e" gracePeriod=30
Sep 11 14:26:03.789640 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.789608    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:03.838189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.838151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"]
Sep 11 14:26:03.838391 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.838368    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj" podUID="1baf3c75-83f2-47bb-a303-5372130fdb06" containerName="payload-processing" containerID="cri-o://d04b00be8299108abffa21ff3ded02d69cf38a9f4036c45ae8a43af53a0ae2f0" gracePeriod=30
Sep 11 14:26:03.847857 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.847818    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"]
Sep 11 14:26:03.848094 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.848067    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g" podUID="44396958-0b5f-4721-a102-0d8a9baa3063" containerName="payload-pre-processing" containerID="cri-o://1e2b69ec1e929df5e8fb26d2a5d3770acbc5ca31fc7f3f7dbb8527c169cbc7b7" gracePeriod=30
Sep 11 14:26:03.903912 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:03.903876    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:04.006308 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.006263    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:04.094783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.094758    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:26:04.100758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.100704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:04.129120 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.129067    2558 generic.go:356] "Generic (PLEG): container finished" podID="44396958-0b5f-4721-a102-0d8a9baa3063" containerID="1e2b69ec1e929df5e8fb26d2a5d3770acbc5ca31fc7f3f7dbb8527c169cbc7b7" exitCode=0
Sep 11 14:26:04.129268 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.129141    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g" event={"ID":"44396958-0b5f-4721-a102-0d8a9baa3063","Type":"ContainerDied","Data":"1e2b69ec1e929df5e8fb26d2a5d3770acbc5ca31fc7f3f7dbb8527c169cbc7b7"}
Sep 11 14:26:04.131217 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.131191    2558 generic.go:356] "Generic (PLEG): container finished" podID="76af003b-80b2-4d8c-8914-3e9b2c307a0c" containerID="3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e" exitCode=0
Sep 11 14:26:04.131341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.131240    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"
Sep 11 14:26:04.131341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.131273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs" event={"ID":"76af003b-80b2-4d8c-8914-3e9b2c307a0c","Type":"ContainerDied","Data":"3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e"}
Sep 11 14:26:04.131341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.131321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs" event={"ID":"76af003b-80b2-4d8c-8914-3e9b2c307a0c","Type":"ContainerDied","Data":"a79821b0a81f72df550f14ae78c9aa7e399b3890087c10f3ff771607eba75920"}
Sep 11 14:26:04.131519 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.131350    2558 scope.go:117] "RemoveContainer" containerID="3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e"
Sep 11 14:26:04.132701 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.132677    2558 generic.go:356] "Generic (PLEG): container finished" podID="1baf3c75-83f2-47bb-a303-5372130fdb06" containerID="d04b00be8299108abffa21ff3ded02d69cf38a9f4036c45ae8a43af53a0ae2f0" exitCode=0
Sep 11 14:26:04.132819 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.132742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj" event={"ID":"1baf3c75-83f2-47bb-a303-5372130fdb06","Type":"ContainerDied","Data":"d04b00be8299108abffa21ff3ded02d69cf38a9f4036c45ae8a43af53a0ae2f0"}
Sep 11 14:26:04.132894 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.132823    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:26:04.142976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.142951    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:26:04.145040 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.144995    2558 scope.go:117] "RemoveContainer" containerID="3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e"
Sep 11 14:26:04.145356 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:26:04.145329    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e\": container with ID starting with 3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e not found: ID does not exist" containerID="3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e"
Sep 11 14:26:04.145436 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.145368    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e"} err="failed to get container status \"3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e\": rpc error: code = NotFound desc = could not find container \"3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e\": container with ID starting with 3b4e1d8083d7c284e4831aeff5817f8b4cf269c9ab668ac7b79c1e6ae9647b6e not found: ID does not exist"
Sep 11 14:26:04.212821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.212789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4stv\" (UniqueName: \"kubernetes.io/projected/1baf3c75-83f2-47bb-a303-5372130fdb06-kube-api-access-m4stv\") pod \"1baf3c75-83f2-47bb-a303-5372130fdb06\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") "
Sep 11 14:26:04.212984 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.212862    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9bjj\" (UniqueName: \"kubernetes.io/projected/76af003b-80b2-4d8c-8914-3e9b2c307a0c-kube-api-access-j9bjj\") pod \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") "
Sep 11 14:26:04.212984 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.212953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baf3c75-83f2-47bb-a303-5372130fdb06-plugins-config-volume\") pod \"1baf3c75-83f2-47bb-a303-5372130fdb06\" (UID: \"1baf3c75-83f2-47bb-a303-5372130fdb06\") "
Sep 11 14:26:04.213115 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213068    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76af003b-80b2-4d8c-8914-3e9b2c307a0c-openshift-service-ca\") pod \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") "
Sep 11 14:26:04.213115 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-metrics-certs\") pod \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") "
Sep 11 14:26:04.213229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213147    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-maas-api-tls\") pod \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\" (UID: \"76af003b-80b2-4d8c-8914-3e9b2c307a0c\") "
Sep 11 14:26:04.213229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213182    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv5dx\" (UniqueName: \"kubernetes.io/projected/44396958-0b5f-4721-a102-0d8a9baa3063-kube-api-access-qv5dx\") pod \"44396958-0b5f-4721-a102-0d8a9baa3063\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") "
Sep 11 14:26:04.213229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213217    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44396958-0b5f-4721-a102-0d8a9baa3063-plugins-config-volume\") pod \"44396958-0b5f-4721-a102-0d8a9baa3063\" (UID: \"44396958-0b5f-4721-a102-0d8a9baa3063\") "
Sep 11 14:26:04.213438 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213402    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1baf3c75-83f2-47bb-a303-5372130fdb06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1baf3c75-83f2-47bb-a303-5372130fdb06" (UID: "1baf3c75-83f2-47bb-a303-5372130fdb06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:26:04.213576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213439    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76af003b-80b2-4d8c-8914-3e9b2c307a0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76af003b-80b2-4d8c-8914-3e9b2c307a0c" (UID: "76af003b-80b2-4d8c-8914-3e9b2c307a0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:26:04.213802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213754    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baf3c75-83f2-47bb-a303-5372130fdb06-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.213802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213780    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76af003b-80b2-4d8c-8914-3e9b2c307a0c-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.213802 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.213794    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44396958-0b5f-4721-a102-0d8a9baa3063-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44396958-0b5f-4721-a102-0d8a9baa3063" (UID: "44396958-0b5f-4721-a102-0d8a9baa3063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:26:04.215238 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.215213    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76af003b-80b2-4d8c-8914-3e9b2c307a0c-kube-api-access-j9bjj" (OuterVolumeSpecName: "kube-api-access-j9bjj") pod "76af003b-80b2-4d8c-8914-3e9b2c307a0c" (UID: "76af003b-80b2-4d8c-8914-3e9b2c307a0c"). InnerVolumeSpecName "kube-api-access-j9bjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:04.215340 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.215241    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76af003b-80b2-4d8c-8914-3e9b2c307a0c" (UID: "76af003b-80b2-4d8c-8914-3e9b2c307a0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:26:04.215400 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.215334    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76af003b-80b2-4d8c-8914-3e9b2c307a0c" (UID: "76af003b-80b2-4d8c-8914-3e9b2c307a0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:26:04.215444 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.215423    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baf3c75-83f2-47bb-a303-5372130fdb06-kube-api-access-m4stv" (OuterVolumeSpecName: "kube-api-access-m4stv") pod "1baf3c75-83f2-47bb-a303-5372130fdb06" (UID: "1baf3c75-83f2-47bb-a303-5372130fdb06"). InnerVolumeSpecName "kube-api-access-m4stv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:04.215675 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.215658    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44396958-0b5f-4721-a102-0d8a9baa3063-kube-api-access-qv5dx" (OuterVolumeSpecName: "kube-api-access-qv5dx") pod "44396958-0b5f-4721-a102-0d8a9baa3063" (UID: "44396958-0b5f-4721-a102-0d8a9baa3063"). InnerVolumeSpecName "kube-api-access-qv5dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:04.315109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.314991    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4stv\" (UniqueName: \"kubernetes.io/projected/1baf3c75-83f2-47bb-a303-5372130fdb06-kube-api-access-m4stv\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.315109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.315021    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9bjj\" (UniqueName: \"kubernetes.io/projected/76af003b-80b2-4d8c-8914-3e9b2c307a0c-kube-api-access-j9bjj\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.315109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.315061    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.315109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.315073    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76af003b-80b2-4d8c-8914-3e9b2c307a0c-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.315109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.315084    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5dx\" (UniqueName: \"kubernetes.io/projected/44396958-0b5f-4721-a102-0d8a9baa3063-kube-api-access-qv5dx\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.315109 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.315098    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44396958-0b5f-4721-a102-0d8a9baa3063-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:04.468149 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.464362    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"]
Sep 11 14:26:04.469804 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.469779    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfa49a-77cfbc7ffd-68cvs"]
Sep 11 14:26:04.503480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.503445    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76af003b-80b2-4d8c-8914-3e9b2c307a0c" path="/var/lib/kubelet/pods/76af003b-80b2-4d8c-8914-3e9b2c307a0c/volumes"
Sep 11 14:26:04.706318 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.706277    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:04.810692 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:04.810655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:05.138744 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.138650    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"
Sep 11 14:26:05.138744 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.138655    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g" event={"ID":"44396958-0b5f-4721-a102-0d8a9baa3063","Type":"ContainerDied","Data":"f9209965b1a6a7d58efe39a3dbeb60da893cc6448d85f70e46813e2023e1540b"}
Sep 11 14:26:05.138744 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.138725    2558 scope.go:117] "RemoveContainer" containerID="1e2b69ec1e929df5e8fb26d2a5d3770acbc5ca31fc7f3f7dbb8527c169cbc7b7"
Sep 11 14:26:05.141249 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.141221    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj" event={"ID":"1baf3c75-83f2-47bb-a303-5372130fdb06","Type":"ContainerDied","Data":"8c75995478f9ad991ec98dc3358da468a58beeedd72e71778962bcbb3e432644"}
Sep 11 14:26:05.141385 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.141270    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"
Sep 11 14:26:05.147602 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.147580    2558 scope.go:117] "RemoveContainer" containerID="d04b00be8299108abffa21ff3ded02d69cf38a9f4036c45ae8a43af53a0ae2f0"
Sep 11 14:26:05.172911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.172878    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"]
Sep 11 14:26:05.180271 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.180244    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bfa49a-64946bc85d-sg69g"]
Sep 11 14:26:05.195868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.195823    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"]
Sep 11 14:26:05.198588 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.198560    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bfa49a-67f9c4f799-l6rdj"]
Sep 11 14:26:05.705746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:05.705713    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:06.504252 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:06.504220    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1baf3c75-83f2-47bb-a303-5372130fdb06" path="/var/lib/kubelet/pods/1baf3c75-83f2-47bb-a303-5372130fdb06/volumes"
Sep 11 14:26:06.504638 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:06.504594    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44396958-0b5f-4721-a102-0d8a9baa3063" path="/var/lib/kubelet/pods/44396958-0b5f-4721-a102-0d8a9baa3063/volumes"
Sep 11 14:26:06.780334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:06.780234    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:07.666934 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:07.666899    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:08.506514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:08.506472    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:10.950920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:10.950883    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:11.052547 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:11.052503    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:11.274859 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:11.274766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:11.777464 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:11.777431    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:12.862074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:12.862014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:14.004975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:14.004934    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:14.824903 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:14.824870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:15.733673 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:15.733629    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:16.603309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:16.603277    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:20.184973 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:20.184932    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:26:22.372667 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.372619    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:22.421621 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.421588    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"]
Sep 11 14:26:22.421877 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.421852    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh" podUID="b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" containerName="payload-processing" containerID="cri-o://563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6" gracePeriod=30
Sep 11 14:26:22.480587 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.480521    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:22.588702 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.588665    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:22.676936 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.676900    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:26:22.798267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.798235    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:22.799156 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.799132    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-plugins-config-volume\") pod \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") "
Sep 11 14:26:22.799303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.799201    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phtpf\" (UniqueName: \"kubernetes.io/projected/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-kube-api-access-phtpf\") pod \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\" (UID: \"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7\") "
Sep 11 14:26:22.799500 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.799472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" (UID: "b7ecb7e6-9369-4606-97c9-f18f93aa9ba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:26:22.801438 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.801413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-kube-api-access-phtpf" (OuterVolumeSpecName: "kube-api-access-phtpf") pod "b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" (UID: "b7ecb7e6-9369-4606-97c9-f18f93aa9ba7"). InnerVolumeSpecName "kube-api-access-phtpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:22.900631 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.900581    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:22.900631 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:22.900620    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtpf\" (UniqueName: \"kubernetes.io/projected/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7-kube-api-access-phtpf\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:23.081082 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.081021    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:23.231943 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.231911    2558 generic.go:356] "Generic (PLEG): container finished" podID="b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" containerID="563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6" exitCode=0
Sep 11 14:26:23.232163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.231954    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"
Sep 11 14:26:23.232163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.231991    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh" event={"ID":"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7","Type":"ContainerDied","Data":"563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6"}
Sep 11 14:26:23.232163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.232063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh" event={"ID":"b7ecb7e6-9369-4606-97c9-f18f93aa9ba7","Type":"ContainerDied","Data":"9e75104ad27da8262814fe972a6f1626230700b7ff0f41edd5ab87bb3916168e"}
Sep 11 14:26:23.232163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.232089    2558 scope.go:117] "RemoveContainer" containerID="563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6"
Sep 11 14:26:23.241193 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.241173    2558 scope.go:117] "RemoveContainer" containerID="563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6"
Sep 11 14:26:23.241487 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:26:23.241463    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6\": container with ID starting with 563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6 not found: ID does not exist" containerID="563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6"
Sep 11 14:26:23.241588 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.241492    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6"} err="failed to get container status \"563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6\": rpc error: code = NotFound desc = could not find container \"563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6\": container with ID starting with 563a1423e60a1b4ad355ee7de8894ae002a8dfcbf17693b5c97b6c3b905e8da6 not found: ID does not exist"
Sep 11 14:26:23.255799 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.255776    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"]
Sep 11 14:26:23.260475 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.260455    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb993c-6c8ff4c9df-hskhh"]
Sep 11 14:26:23.265220 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:23.265201    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:24.130600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:24.130566    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:24.504126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:24.504094    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" path="/var/lib/kubelet/pods/b7ecb7e6-9369-4606-97c9-f18f93aa9ba7/volumes"
Sep 11 14:26:25.308653 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.308616    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:25.350153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350120    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7708-832f35"]
Sep 11 14:26:25.350501 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350489    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" containerName="payload-processing"
Sep 11 14:26:25.350545 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350503    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" containerName="payload-processing"
Sep 11 14:26:25.350545 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350517    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44396958-0b5f-4721-a102-0d8a9baa3063" containerName="payload-pre-processing"
Sep 11 14:26:25.350545 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350523    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="44396958-0b5f-4721-a102-0d8a9baa3063" containerName="payload-pre-processing"
Sep 11 14:26:25.350545 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350536    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1baf3c75-83f2-47bb-a303-5372130fdb06" containerName="payload-processing"
Sep 11 14:26:25.350545 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350541    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1baf3c75-83f2-47bb-a303-5372130fdb06" containerName="payload-processing"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350548    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76af003b-80b2-4d8c-8914-3e9b2c307a0c" containerName="maas-api"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350553    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="76af003b-80b2-4d8c-8914-3e9b2c307a0c" containerName="maas-api"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350561    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151fdb49-9ea5-4787-ae20-80ed9405f360" containerName="payload-pre-processing"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350566    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="151fdb49-9ea5-4787-ae20-80ed9405f360" containerName="payload-pre-processing"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350625    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1baf3c75-83f2-47bb-a303-5372130fdb06" containerName="payload-processing"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350634    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="76af003b-80b2-4d8c-8914-3e9b2c307a0c" containerName="maas-api"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350639    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="151fdb49-9ea5-4787-ae20-80ed9405f360" containerName="payload-pre-processing"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350645    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="44396958-0b5f-4721-a102-0d8a9baa3063" containerName="payload-pre-processing"
Sep 11 14:26:25.350695 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.350650    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7ecb7e6-9369-4606-97c9-f18f93aa9ba7" containerName="payload-processing"
Sep 11 14:26:25.355068 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.355048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:25.358533 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.358509    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4lz74\""
Sep 11 14:26:25.359798 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.359774    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-832f35"]
Sep 11 14:26:25.423543 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.423501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pmc\" (UniqueName: \"kubernetes.io/projected/12f94d64-4ac1-477c-817d-29b5f73478d6-kube-api-access-d4pmc\") pod \"test-curl-7708-832f35\" (UID: \"12f94d64-4ac1-477c-817d-29b5f73478d6\") " pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:25.523969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.523932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pmc\" (UniqueName: \"kubernetes.io/projected/12f94d64-4ac1-477c-817d-29b5f73478d6-kube-api-access-d4pmc\") pod \"test-curl-7708-832f35\" (UID: \"12f94d64-4ac1-477c-817d-29b5f73478d6\") " pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:25.547686 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.547653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pmc\" (UniqueName: \"kubernetes.io/projected/12f94d64-4ac1-477c-817d-29b5f73478d6-kube-api-access-d4pmc\") pod \"test-curl-7708-832f35\" (UID: \"12f94d64-4ac1-477c-817d-29b5f73478d6\") " pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:25.665813 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.665709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:25.945016 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:25.944992    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-832f35"]
Sep 11 14:26:25.946574 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:26:25.946546    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12f94d64_4ac1_477c_817d_29b5f73478d6.slice/crio-b15fc9abbd25e278741eddde5c77f479ba4174d17cb77d511140789f9edc1994 WatchSource:0}: Error finding container b15fc9abbd25e278741eddde5c77f479ba4174d17cb77d511140789f9edc1994: Status 404 returned error can't find the container with id b15fc9abbd25e278741eddde5c77f479ba4174d17cb77d511140789f9edc1994
Sep 11 14:26:26.248963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:26.248864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-832f35" event={"ID":"12f94d64-4ac1-477c-817d-29b5f73478d6","Type":"ContainerStarted","Data":"a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc"}
Sep 11 14:26:26.248963 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:26.248913    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-832f35" event={"ID":"12f94d64-4ac1-477c-817d-29b5f73478d6","Type":"ContainerStarted","Data":"b15fc9abbd25e278741eddde5c77f479ba4174d17cb77d511140789f9edc1994"}
Sep 11 14:26:26.274614 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:26.274559    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7708-832f35" podStartSLOduration=1.27454355 podStartE2EDuration="1.27454355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:26.273082069 +0000 UTC m=+1720.236697040" watchObservedRunningTime="2026-09-11 14:26:26.27454355 +0000 UTC m=+1720.238158574"
Sep 11 14:26:26.318330 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:26.318294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:26.942016 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:26.941977    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7708-832f35"]
Sep 11 14:26:26.947284 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:26.947251    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7708-832f35"]
Sep 11 14:26:27.253194 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:26:27.253164    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7708-832f35" container="test-curl-7708-832f35"
Sep 11 14:26:28.257786 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:28.257715    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7708-832f35" podUID="12f94d64-4ac1-477c-817d-29b5f73478d6" containerName="test-curl-7708-832f35" containerID="cri-o://a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc" gracePeriod=2
Sep 11 14:26:28.260112 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:26:28.260082    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-832f35\" is forbidden: User \"system:node:ip-10-0-140-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-144.ec2.internal' and this object" podUID="12f94d64-4ac1-477c-817d-29b5f73478d6" pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:29.125065 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:29.125000    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"]
Sep 11 14:26:29.165117 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:29.165078    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:30.126437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.126397    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"]
Sep 11 14:26:30.171655 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.171615    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:30.358490 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.358457    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:30.443416 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.443393    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:30.464844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.464815    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4pmc\" (UniqueName: \"kubernetes.io/projected/12f94d64-4ac1-477c-817d-29b5f73478d6-kube-api-access-d4pmc\") pod \"12f94d64-4ac1-477c-817d-29b5f73478d6\" (UID: \"12f94d64-4ac1-477c-817d-29b5f73478d6\") "
Sep 11 14:26:30.466748 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.466718    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f94d64-4ac1-477c-817d-29b5f73478d6-kube-api-access-d4pmc" (OuterVolumeSpecName: "kube-api-access-d4pmc") pod "12f94d64-4ac1-477c-817d-29b5f73478d6" (UID: "12f94d64-4ac1-477c-817d-29b5f73478d6"). InnerVolumeSpecName "kube-api-access-d4pmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:30.483605 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.483580    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:30.503350 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.503325    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f94d64-4ac1-477c-817d-29b5f73478d6" path="/var/lib/kubelet/pods/12f94d64-4ac1-477c-817d-29b5f73478d6/volumes"
Sep 11 14:26:30.566563 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.566538    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pmc\" (UniqueName: \"kubernetes.io/projected/12f94d64-4ac1-477c-817d-29b5f73478d6-kube-api-access-d4pmc\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:30.696241 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:30.696215    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:31.274318 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:31.274284    2558 generic.go:356] "Generic (PLEG): container finished" podID="12f94d64-4ac1-477c-817d-29b5f73478d6" containerID="a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc" exitCode=137
Sep 11 14:26:31.274770 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:31.274320    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-832f35"
Sep 11 14:26:31.274770 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:31.274383    2558 scope.go:117] "RemoveContainer" containerID="a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc"
Sep 11 14:26:31.283314 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:31.283296    2558 scope.go:117] "RemoveContainer" containerID="a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc"
Sep 11 14:26:31.283657 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:26:31.283635    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc\": container with ID starting with a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc not found: ID does not exist" containerID="a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc"
Sep 11 14:26:31.283758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:31.283666    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc"} err="failed to get container status \"a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc\": rpc error: code = NotFound desc = could not find container \"a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc\": container with ID starting with a516fbe86c918e888ed8c286c468b18565a5d4b78aae6957c896119cd9cfd5fc not found: ID does not exist"
Sep 11 14:26:31.332561 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:31.332524    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:41.280951 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:41.280914    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:26:42.041264 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:42.041214    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:43.164432 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:43.164397    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:43.967390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:43.967356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:44.144017 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.143982    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj" podUID="9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" containerName="main" containerID="cri-o://e90ab570e0498612dd8ffaf21059c5323f96d0894a5d834fb03f3d0e04510331" gracePeriod=45
Sep 11 14:26:44.339215 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.339182    2558 generic.go:356] "Generic (PLEG): container finished" podID="9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" containerID="e90ab570e0498612dd8ffaf21059c5323f96d0894a5d834fb03f3d0e04510331" exitCode=0
Sep 11 14:26:44.339615 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.339265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj" event={"ID":"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac","Type":"ContainerDied","Data":"e90ab570e0498612dd8ffaf21059c5323f96d0894a5d834fb03f3d0e04510331"}
Sep 11 14:26:44.397395 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.397364    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:26:44.491239 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491201    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-dshm\") pod \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") "
Sep 11 14:26:44.491414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491260    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tmp-dir\") pod \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") "
Sep 11 14:26:44.491414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-model-cache\") pod \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") "
Sep 11 14:26:44.491414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491325    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-home\") pod \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") "
Sep 11 14:26:44.491414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-kube-api-access-58mdj\") pod \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") "
Sep 11 14:26:44.491586 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tls-certs\") pod \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\" (UID: \"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac\") "
Sep 11 14:26:44.491644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491614    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-home" (OuterVolumeSpecName: "home") pod "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" (UID: "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:44.491708 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491622    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" (UID: "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:44.491708 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491637    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" (UID: "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:44.491821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491740    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:44.491821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491752    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:44.491821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.491763    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:44.493475 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.493445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-kube-api-access-58mdj" (OuterVolumeSpecName: "kube-api-access-58mdj") pod "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" (UID: "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac"). InnerVolumeSpecName "kube-api-access-58mdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:44.493580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.493453    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-dshm" (OuterVolumeSpecName: "dshm") pod "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" (UID: "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:44.493580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.493485    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" (UID: "9a4e27e4-fdbd-47cd-a4ae-3088c92237ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:26:44.592906 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.592874    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:44.592906 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.592907    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mdj\" (UniqueName: \"kubernetes.io/projected/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-kube-api-access-58mdj\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:44.593140 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.592922    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:44.631846 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.631810    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:44.690212 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.690173    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"]
Sep 11 14:26:44.690412 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.690388    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq" podUID="cc7946ec-d010-4180-8246-1245e5c6393a" containerName="payload-pre-processing" containerID="cri-o://33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963" gracePeriod=30
Sep 11 14:26:44.734203 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.734174    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:44.834453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.834295    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:44.910720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.910693    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:44.925170 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.925149    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:26:44.997523 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.997494    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7946ec-d010-4180-8246-1245e5c6393a-plugins-config-volume\") pod \"cc7946ec-d010-4180-8246-1245e5c6393a\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") "
Sep 11 14:26:44.997698 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.997579    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nshwz\" (UniqueName: \"kubernetes.io/projected/cc7946ec-d010-4180-8246-1245e5c6393a-kube-api-access-nshwz\") pod \"cc7946ec-d010-4180-8246-1245e5c6393a\" (UID: \"cc7946ec-d010-4180-8246-1245e5c6393a\") "
Sep 11 14:26:44.997923 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.997886    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7946ec-d010-4180-8246-1245e5c6393a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc7946ec-d010-4180-8246-1245e5c6393a" (UID: "cc7946ec-d010-4180-8246-1245e5c6393a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:26:44.999585 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:44.999562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7946ec-d010-4180-8246-1245e5c6393a-kube-api-access-nshwz" (OuterVolumeSpecName: "kube-api-access-nshwz") pod "cc7946ec-d010-4180-8246-1245e5c6393a" (UID: "cc7946ec-d010-4180-8246-1245e5c6393a"). InnerVolumeSpecName "kube-api-access-nshwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:45.099181 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.099092    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nshwz\" (UniqueName: \"kubernetes.io/projected/cc7946ec-d010-4180-8246-1245e5c6393a-kube-api-access-nshwz\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.099181 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.099123    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7946ec-d010-4180-8246-1245e5c6393a-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.144879 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.144841    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94" podUID="e488e71e-8662-4e35-8263-b4e2990b2d39" containerName="main" containerID="cri-o://30dae9c9b0ffdef5ffee60fa735d76f91f3e5fc7ae813c8725e4c1265b320865" gracePeriod=45
Sep 11 14:26:45.342236 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.342199    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:45.347659 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.347629    2558 generic.go:356] "Generic (PLEG): container finished" podID="e488e71e-8662-4e35-8263-b4e2990b2d39" containerID="30dae9c9b0ffdef5ffee60fa735d76f91f3e5fc7ae813c8725e4c1265b320865" exitCode=0
Sep 11 14:26:45.347793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.347703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94" event={"ID":"e488e71e-8662-4e35-8263-b4e2990b2d39","Type":"ContainerDied","Data":"30dae9c9b0ffdef5ffee60fa735d76f91f3e5fc7ae813c8725e4c1265b320865"}
Sep 11 14:26:45.349333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.349273    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"
Sep 11 14:26:45.349333 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.349312    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj" event={"ID":"9a4e27e4-fdbd-47cd-a4ae-3088c92237ac","Type":"ContainerDied","Data":"53c105525d2543369f41a2aa2324f4e4c954b201f1038d20ef8761b28f78b889"}
Sep 11 14:26:45.349508 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.349368    2558 scope.go:117] "RemoveContainer" containerID="e90ab570e0498612dd8ffaf21059c5323f96d0894a5d834fb03f3d0e04510331"
Sep 11 14:26:45.350975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.350949    2558 generic.go:356] "Generic (PLEG): container finished" podID="cc7946ec-d010-4180-8246-1245e5c6393a" containerID="33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963" exitCode=0
Sep 11 14:26:45.351100 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.350979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq" event={"ID":"cc7946ec-d010-4180-8246-1245e5c6393a","Type":"ContainerDied","Data":"33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963"}
Sep 11 14:26:45.351100 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.351012    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"
Sep 11 14:26:45.351100 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.351075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq" event={"ID":"cc7946ec-d010-4180-8246-1245e5c6393a","Type":"ContainerDied","Data":"1ad8f18229c0c04a376e7f7574b03ef6557ed7ce7801dede4a204f32bbe06e68"}
Sep 11 14:26:45.362822 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.362734    2558 scope.go:117] "RemoveContainer" containerID="33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963"
Sep 11 14:26:45.371186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.371166    2558 scope.go:117] "RemoveContainer" containerID="33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963"
Sep 11 14:26:45.371515 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:26:45.371492    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963\": container with ID starting with 33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963 not found: ID does not exist" containerID="33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963"
Sep 11 14:26:45.371600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.371528    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963"} err="failed to get container status \"33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963\": rpc error: code = NotFound desc = could not find container \"33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963\": container with ID starting with 33e39d2c566896fd075d9b9c22cd4535d1b99bd613e4b8bcf12a1672fcc76963 not found: ID does not exist"
Sep 11 14:26:45.387855 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.387821    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"]
Sep 11 14:26:45.391590 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.391568    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/sub-test-model-1e2e7d-kserve-668457b7f9-68ftj"]
Sep 11 14:26:45.405406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.405383    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"]
Sep 11 14:26:45.406381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.406362    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:26:45.411721 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.411697    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-335a68-7d4b6f6598-tkccq"]
Sep 11 14:26:45.502402 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502366    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-home\") pod \"e488e71e-8662-4e35-8263-b4e2990b2d39\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") "
Sep 11 14:26:45.502402 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502402    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-model-cache\") pod \"e488e71e-8662-4e35-8263-b4e2990b2d39\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") "
Sep 11 14:26:45.502665 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502456    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-dshm\") pod \"e488e71e-8662-4e35-8263-b4e2990b2d39\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") "
Sep 11 14:26:45.502665 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502495    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-tmp-dir\") pod \"e488e71e-8662-4e35-8263-b4e2990b2d39\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") "
Sep 11 14:26:45.502665 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e488e71e-8662-4e35-8263-b4e2990b2d39-tls-certs\") pod \"e488e71e-8662-4e35-8263-b4e2990b2d39\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") "
Sep 11 14:26:45.502665 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502596    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46gft\" (UniqueName: \"kubernetes.io/projected/e488e71e-8662-4e35-8263-b4e2990b2d39-kube-api-access-46gft\") pod \"e488e71e-8662-4e35-8263-b4e2990b2d39\" (UID: \"e488e71e-8662-4e35-8263-b4e2990b2d39\") "
Sep 11 14:26:45.502862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502673    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-home" (OuterVolumeSpecName: "home") pod "e488e71e-8662-4e35-8263-b4e2990b2d39" (UID: "e488e71e-8662-4e35-8263-b4e2990b2d39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:45.502862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-model-cache" (OuterVolumeSpecName: "model-cache") pod "e488e71e-8662-4e35-8263-b4e2990b2d39" (UID: "e488e71e-8662-4e35-8263-b4e2990b2d39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:45.502862 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502794    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e488e71e-8662-4e35-8263-b4e2990b2d39" (UID: "e488e71e-8662-4e35-8263-b4e2990b2d39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:45.502966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502891    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.502966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502903    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.502966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.502931    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.504632 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.504605    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-dshm" (OuterVolumeSpecName: "dshm") pod "e488e71e-8662-4e35-8263-b4e2990b2d39" (UID: "e488e71e-8662-4e35-8263-b4e2990b2d39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:26:45.504746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.504645    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e488e71e-8662-4e35-8263-b4e2990b2d39-kube-api-access-46gft" (OuterVolumeSpecName: "kube-api-access-46gft") pod "e488e71e-8662-4e35-8263-b4e2990b2d39" (UID: "e488e71e-8662-4e35-8263-b4e2990b2d39"). InnerVolumeSpecName "kube-api-access-46gft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:26:45.504746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.504693    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e488e71e-8662-4e35-8263-b4e2990b2d39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e488e71e-8662-4e35-8263-b4e2990b2d39" (UID: "e488e71e-8662-4e35-8263-b4e2990b2d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:26:45.603712 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.603607    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e488e71e-8662-4e35-8263-b4e2990b2d39-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.603712 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.603639    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e488e71e-8662-4e35-8263-b4e2990b2d39-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:45.603712 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:45.603655    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46gft\" (UniqueName: \"kubernetes.io/projected/e488e71e-8662-4e35-8263-b4e2990b2d39-kube-api-access-46gft\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:26:46.064519 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.064482    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:46.357018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.356928    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"
Sep 11 14:26:46.357018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.356952    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94" event={"ID":"e488e71e-8662-4e35-8263-b4e2990b2d39","Type":"ContainerDied","Data":"41211ff61f363f50ae98b979a099cda9760e505cab87928c59b2ab7d319047a8"}
Sep 11 14:26:46.357018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.357004    2558 scope.go:117] "RemoveContainer" containerID="30dae9c9b0ffdef5ffee60fa735d76f91f3e5fc7ae813c8725e4c1265b320865"
Sep 11 14:26:46.391199 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.391170    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"]
Sep 11 14:26:46.409126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.409097    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/sub-test-model-031a56-kserve-7464f84cdc-f5q94"]
Sep 11 14:26:46.503180 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.503147    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" path="/var/lib/kubelet/pods/9a4e27e4-fdbd-47cd-a4ae-3088c92237ac/volumes"
Sep 11 14:26:46.503534 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.503521    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7946ec-d010-4180-8246-1245e5c6393a" path="/var/lib/kubelet/pods/cc7946ec-d010-4180-8246-1245e5c6393a/volumes"
Sep 11 14:26:46.503878 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:46.503867    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e488e71e-8662-4e35-8263-b4e2990b2d39" path="/var/lib/kubelet/pods/e488e71e-8662-4e35-8263-b4e2990b2d39/volumes"
Sep 11 14:26:47.135698 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:47.135603    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:48.862290 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862260    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"]
Sep 11 14:26:48.862663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862627    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" containerName="main"
Sep 11 14:26:48.862663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862639    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" containerName="main"
Sep 11 14:26:48.862663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862649    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e488e71e-8662-4e35-8263-b4e2990b2d39" containerName="main"
Sep 11 14:26:48.862663 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862654    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e488e71e-8662-4e35-8263-b4e2990b2d39" containerName="main"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862671    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f94d64-4ac1-477c-817d-29b5f73478d6" containerName="test-curl-7708-832f35"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862681    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f94d64-4ac1-477c-817d-29b5f73478d6" containerName="test-curl-7708-832f35"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862692    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7946ec-d010-4180-8246-1245e5c6393a" containerName="payload-pre-processing"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862699    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7946ec-d010-4180-8246-1245e5c6393a" containerName="payload-pre-processing"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862759    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7946ec-d010-4180-8246-1245e5c6393a" containerName="payload-pre-processing"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862770    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e488e71e-8662-4e35-8263-b4e2990b2d39" containerName="main"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862777    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a4e27e4-fdbd-47cd-a4ae-3088c92237ac" containerName="main"
Sep 11 14:26:48.862805 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.862784    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f94d64-4ac1-477c-817d-29b5f73478d6" containerName="test-curl-7708-832f35"
Sep 11 14:26:48.865827 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.865807    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:48.870976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.870947    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"rate-test-model-1e2e7d-kserve-self-signed-certs\""
Sep 11 14:26:48.871187 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.871018    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"default-dockercfg-zvrmr\""
Sep 11 14:26:48.871187 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.870962    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"kube-root-ca.crt\""
Sep 11 14:26:48.871187 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.871065    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1e2e7d\"/\"openshift-service-ca.crt\""
Sep 11 14:26:48.881870 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.881839    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"]
Sep 11 14:26:48.917486 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.917449    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:48.935499 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.935466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-model-cache\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:48.935644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.935512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451897d1-e0d0-4146-85d7-e8f410f17c71-tls-certs\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:48.935644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.935573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-tmp-dir\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:48.935644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.935595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-dshm\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:48.935644 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.935622    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrn4\" (UniqueName: \"kubernetes.io/projected/451897d1-e0d0-4146-85d7-e8f410f17c71-kube-api-access-vsrn4\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:48.935781 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:48.935644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-home\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrn4\" (UniqueName: \"kubernetes.io/projected/451897d1-e0d0-4146-85d7-e8f410f17c71-kube-api-access-vsrn4\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-home\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036779 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-model-cache\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036860 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451897d1-e0d0-4146-85d7-e8f410f17c71-tls-certs\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036923 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-tmp-dir\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-dshm\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.036976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.036946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-home\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.037117 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.037086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-model-cache\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.037207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.037185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-tmp-dir\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.039074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.039055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-dshm\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.039415 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.039396    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451897d1-e0d0-4146-85d7-e8f410f17c71-tls-certs\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.048508 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.048489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrn4\" (UniqueName: \"kubernetes.io/projected/451897d1-e0d0-4146-85d7-e8f410f17c71-kube-api-access-vsrn4\") pod \"rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") " pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.174706 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.174627    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:49.452188 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:49.452157    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"]
Sep 11 14:26:49.455243 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:26:49.455218    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod451897d1_e0d0_4146_85d7_e8f410f17c71.slice/crio-601c372d664d96cdd487074a63556dd09b7f2e6a587f02b55a92f7cd84057f63 WatchSource:0}: Error finding container 601c372d664d96cdd487074a63556dd09b7f2e6a587f02b55a92f7cd84057f63: Status 404 returned error can't find the container with id 601c372d664d96cdd487074a63556dd09b7f2e6a587f02b55a92f7cd84057f63
Sep 11 14:26:50.380241 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:50.380198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn" event={"ID":"451897d1-e0d0-4146-85d7-e8f410f17c71","Type":"ContainerStarted","Data":"882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c"}
Sep 11 14:26:50.380241 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:50.380248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn" event={"ID":"451897d1-e0d0-4146-85d7-e8f410f17c71","Type":"ContainerStarted","Data":"601c372d664d96cdd487074a63556dd09b7f2e6a587f02b55a92f7cd84057f63"}
Sep 11 14:26:50.404382 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:50.404330    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn" podStartSLOduration=2.404311931 podStartE2EDuration="2.404311931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:50.403416521 +0000 UTC m=+1744.367031496" watchObservedRunningTime="2026-09-11 14:26:50.404311931 +0000 UTC m=+1744.367926905"
Sep 11 14:26:50.578649 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:50.578611    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:26:52.494865 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:52.494829    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:52.603818 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:52.603781    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:53.031571 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:53.031532    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:26:59.175050 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:59.174995    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:59.175454 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:59.175066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:59.187512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:59.187486    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:26:59.433547 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:26:59.433471    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:27:31.835739 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:31.835701    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:27:39.600360 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.600282    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"]
Sep 11 14:27:39.603971 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.603948    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.607066 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.607005    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"kube-root-ca.crt\""
Sep 11 14:27:39.608197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.608158    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"rate-test-model-031a56-kserve-self-signed-certs\""
Sep 11 14:27:39.608197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.608175    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"openshift-service-ca.crt\""
Sep 11 14:27:39.608350 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.608200    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-031a56\"/\"default-dockercfg-fx44k\""
Sep 11 14:27:39.616088 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.616066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"]
Sep 11 14:27:39.616687 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.616666    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:27:39.669462 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.669435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-dshm\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.669575 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.669480    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-home\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.669575 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.669508    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tls-certs\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.669654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.669574    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-model-cache\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.669654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.669609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tmp-dir\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.669719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.669682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-kube-api-access-bc9jw\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.770633 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.770604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-home\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.770783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.770640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tls-certs\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.770783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.770669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-model-cache\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.770783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.770692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tmp-dir\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.770783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.770758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-kube-api-access-bc9jw\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.771021 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.770801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-dshm\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.771127 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.771017    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-home\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.771127 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.771088    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-model-cache\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.771207 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.771147    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tmp-dir\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.773041 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.773004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-dshm\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.773254 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.773235    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tls-certs\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.778965 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.778945    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-kube-api-access-bc9jw\") pod \"rate-test-model-031a56-kserve-6c554c8d5f-9wqvs\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") " pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:39.913748 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:39.913680    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:40.196342 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:40.196319    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"]
Sep 11 14:27:40.198562 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:27:40.198531    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca73d7cf_268f_46c2_a3a0_3c1c1552a725.slice/crio-f000358ad4e50d07510072a995e534dc1ccdfdcb1e17f2008b05b5acc0ede629 WatchSource:0}: Error finding container f000358ad4e50d07510072a995e534dc1ccdfdcb1e17f2008b05b5acc0ede629: Status 404 returned error can't find the container with id f000358ad4e50d07510072a995e534dc1ccdfdcb1e17f2008b05b5acc0ede629
Sep 11 14:27:40.605691 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:40.605599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs" event={"ID":"ca73d7cf-268f-46c2-a3a0-3c1c1552a725","Type":"ContainerStarted","Data":"677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767"}
Sep 11 14:27:40.605691 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:40.605641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs" event={"ID":"ca73d7cf-268f-46c2-a3a0-3c1c1552a725","Type":"ContainerStarted","Data":"f000358ad4e50d07510072a995e534dc1ccdfdcb1e17f2008b05b5acc0ede629"}
Sep 11 14:27:40.636297 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:40.636245    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs" podStartSLOduration=1.6362314580000001 podStartE2EDuration="1.636231458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:40.634505883 +0000 UTC m=+1794.598120859" watchObservedRunningTime="2026-09-11 14:27:40.636231458 +0000 UTC m=+1794.599846446"
Sep 11 14:27:43.154508 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:43.154466    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:27:46.561202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:46.561165    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:27:46.564431 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:46.564409    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:27:49.914469 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:49.914433    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:49.914970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:49.914524    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:49.927224 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:49.927200    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:50.668396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:50.668366    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:27:56.857720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:27:56.857689    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:28:06.784194 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:06.784154    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:28:12.212454 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:12.212423    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"]
Sep 11 14:28:12.236918 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:12.236884    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:13.151579 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:13.151542    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"]
Sep 11 14:28:13.172646 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:13.172618    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:13.688362 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:13.688333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:14.002488 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:14.002447    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:14.223289 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:14.223255    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:16.641796 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:16.641766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:17.339621 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:17.339587    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:17.489982 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:17.489947    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:17.519377 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:17.519347    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:28:17.829716 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:17.829682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:18.655479 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:18.655440    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:19.560789 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:19.560762    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:19.658257 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:19.658234    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:19.758393 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:19.758367    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:20.060019 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:20.059988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:20.159194 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:20.159165    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:27.230719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.230682    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn" podUID="451897d1-e0d0-4146-85d7-e8f410f17c71" containerName="main" containerID="cri-o://882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c" gracePeriod=45
Sep 11 14:28:27.477900 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.477877    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:28:27.573764 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.573697    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsrn4\" (UniqueName: \"kubernetes.io/projected/451897d1-e0d0-4146-85d7-e8f410f17c71-kube-api-access-vsrn4\") pod \"451897d1-e0d0-4146-85d7-e8f410f17c71\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") "
Sep 11 14:28:27.573764 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.573757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-tmp-dir\") pod \"451897d1-e0d0-4146-85d7-e8f410f17c71\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") "
Sep 11 14:28:27.573972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.573781    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-home\") pod \"451897d1-e0d0-4146-85d7-e8f410f17c71\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") "
Sep 11 14:28:27.573972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.573800    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-dshm\") pod \"451897d1-e0d0-4146-85d7-e8f410f17c71\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") "
Sep 11 14:28:27.573972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.573871    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-model-cache\") pod \"451897d1-e0d0-4146-85d7-e8f410f17c71\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") "
Sep 11 14:28:27.573972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.573891    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451897d1-e0d0-4146-85d7-e8f410f17c71-tls-certs\") pod \"451897d1-e0d0-4146-85d7-e8f410f17c71\" (UID: \"451897d1-e0d0-4146-85d7-e8f410f17c71\") "
Sep 11 14:28:27.574195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.574071    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-home" (OuterVolumeSpecName: "home") pod "451897d1-e0d0-4146-85d7-e8f410f17c71" (UID: "451897d1-e0d0-4146-85d7-e8f410f17c71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:27.574195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.574086    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "451897d1-e0d0-4146-85d7-e8f410f17c71" (UID: "451897d1-e0d0-4146-85d7-e8f410f17c71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:27.574195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.574157    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-model-cache" (OuterVolumeSpecName: "model-cache") pod "451897d1-e0d0-4146-85d7-e8f410f17c71" (UID: "451897d1-e0d0-4146-85d7-e8f410f17c71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:27.575882 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.575853    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451897d1-e0d0-4146-85d7-e8f410f17c71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "451897d1-e0d0-4146-85d7-e8f410f17c71" (UID: "451897d1-e0d0-4146-85d7-e8f410f17c71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:28:27.575984 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.575882    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-dshm" (OuterVolumeSpecName: "dshm") pod "451897d1-e0d0-4146-85d7-e8f410f17c71" (UID: "451897d1-e0d0-4146-85d7-e8f410f17c71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:27.575984 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.575923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451897d1-e0d0-4146-85d7-e8f410f17c71-kube-api-access-vsrn4" (OuterVolumeSpecName: "kube-api-access-vsrn4") pod "451897d1-e0d0-4146-85d7-e8f410f17c71" (UID: "451897d1-e0d0-4146-85d7-e8f410f17c71"). InnerVolumeSpecName "kube-api-access-vsrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:28:27.675367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.675340    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:27.675367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.675358    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:27.675367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.675366    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:27.675518 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.675374    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451897d1-e0d0-4146-85d7-e8f410f17c71-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:27.675518 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.675382    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451897d1-e0d0-4146-85d7-e8f410f17c71-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:27.675518 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.675390    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrn4\" (UniqueName: \"kubernetes.io/projected/451897d1-e0d0-4146-85d7-e8f410f17c71-kube-api-access-vsrn4\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:27.823196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.823172    2558 generic.go:356] "Generic (PLEG): container finished" podID="451897d1-e0d0-4146-85d7-e8f410f17c71" containerID="882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c" exitCode=0
Sep 11 14:28:27.823303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.823213    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"
Sep 11 14:28:27.823303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.823229    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn" event={"ID":"451897d1-e0d0-4146-85d7-e8f410f17c71","Type":"ContainerDied","Data":"882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c"}
Sep 11 14:28:27.823303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.823258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn" event={"ID":"451897d1-e0d0-4146-85d7-e8f410f17c71","Type":"ContainerDied","Data":"601c372d664d96cdd487074a63556dd09b7f2e6a587f02b55a92f7cd84057f63"}
Sep 11 14:28:27.823303 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.823273    2558 scope.go:117] "RemoveContainer" containerID="882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c"
Sep 11 14:28:27.831922 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.831900    2558 scope.go:117] "RemoveContainer" containerID="882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c"
Sep 11 14:28:27.832234 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:28:27.832211    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c\": container with ID starting with 882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c not found: ID does not exist" containerID="882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c"
Sep 11 14:28:27.832295 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.832242    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c"} err="failed to get container status \"882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c\": rpc error: code = NotFound desc = could not find container \"882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c\": container with ID starting with 882322658a7c7e43e3d56d000cbc49657d621eccd2d4125cd71594669547f78c not found: ID does not exist"
Sep 11 14:28:27.845241 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.845219    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"]
Sep 11 14:28:27.850372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:27.850350    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1e2e7d/rate-test-model-1e2e7d-kserve-786c7fbb79-f2jmn"]
Sep 11 14:28:28.170221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.170121    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs" podUID="ca73d7cf-268f-46c2-a3a0-3c1c1552a725" containerName="main" containerID="cri-o://677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767" gracePeriod=45
Sep 11 14:28:28.408134 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.408110    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:28:28.481789 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.481761    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tls-certs\") pod \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") "
Sep 11 14:28:28.481952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.481797    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-model-cache\") pod \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") "
Sep 11 14:28:28.481952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.481831    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tmp-dir\") pod \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") "
Sep 11 14:28:28.481952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.481861    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-dshm\") pod \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") "
Sep 11 14:28:28.482124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.481960    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-home\") pod \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") "
Sep 11 14:28:28.482124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.481990    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-kube-api-access-bc9jw\") pod \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\" (UID: \"ca73d7cf-268f-46c2-a3a0-3c1c1552a725\") "
Sep 11 14:28:28.482240 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.482112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca73d7cf-268f-46c2-a3a0-3c1c1552a725" (UID: "ca73d7cf-268f-46c2-a3a0-3c1c1552a725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:28.482240 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.482122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca73d7cf-268f-46c2-a3a0-3c1c1552a725" (UID: "ca73d7cf-268f-46c2-a3a0-3c1c1552a725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:28.482354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.482265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-home" (OuterVolumeSpecName: "home") pod "ca73d7cf-268f-46c2-a3a0-3c1c1552a725" (UID: "ca73d7cf-268f-46c2-a3a0-3c1c1552a725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:28.482404 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.482385    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:28.482452 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.482403    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:28.482452 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.482417    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:28.483941 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.483917    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca73d7cf-268f-46c2-a3a0-3c1c1552a725" (UID: "ca73d7cf-268f-46c2-a3a0-3c1c1552a725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:28:28.484074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.483988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-dshm" (OuterVolumeSpecName: "dshm") pod "ca73d7cf-268f-46c2-a3a0-3c1c1552a725" (UID: "ca73d7cf-268f-46c2-a3a0-3c1c1552a725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:28:28.484074 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.484054    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-kube-api-access-bc9jw" (OuterVolumeSpecName: "kube-api-access-bc9jw") pod "ca73d7cf-268f-46c2-a3a0-3c1c1552a725" (UID: "ca73d7cf-268f-46c2-a3a0-3c1c1552a725"). InnerVolumeSpecName "kube-api-access-bc9jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:28:28.503735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.503707    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451897d1-e0d0-4146-85d7-e8f410f17c71" path="/var/lib/kubelet/pods/451897d1-e0d0-4146-85d7-e8f410f17c71/volumes"
Sep 11 14:28:28.583301 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.583275    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-kube-api-access-bc9jw\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:28.583301 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.583299    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:28.583436 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.583307    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca73d7cf-268f-46c2-a3a0-3c1c1552a725-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:28:28.830200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.830121    2558 generic.go:356] "Generic (PLEG): container finished" podID="ca73d7cf-268f-46c2-a3a0-3c1c1552a725" containerID="677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767" exitCode=0
Sep 11 14:28:28.830200 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.830182    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"
Sep 11 14:28:28.830396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.830206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs" event={"ID":"ca73d7cf-268f-46c2-a3a0-3c1c1552a725","Type":"ContainerDied","Data":"677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767"}
Sep 11 14:28:28.830396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.830248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs" event={"ID":"ca73d7cf-268f-46c2-a3a0-3c1c1552a725","Type":"ContainerDied","Data":"f000358ad4e50d07510072a995e534dc1ccdfdcb1e17f2008b05b5acc0ede629"}
Sep 11 14:28:28.830396 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.830265    2558 scope.go:117] "RemoveContainer" containerID="677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767"
Sep 11 14:28:28.838138 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.838121    2558 scope.go:117] "RemoveContainer" containerID="677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767"
Sep 11 14:28:28.838403 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:28:28.838383    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767\": container with ID starting with 677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767 not found: ID does not exist" containerID="677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767"
Sep 11 14:28:28.838445 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.838416    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767"} err="failed to get container status \"677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767\": rpc error: code = NotFound desc = could not find container \"677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767\": container with ID starting with 677549e18e3b73373f1b4af9835f436657e71d398402315a0326ca5bd1c81767 not found: ID does not exist"
Sep 11 14:28:28.845743 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.845721    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"]
Sep 11 14:28:28.849786 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:28.849764    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-031a56/rate-test-model-031a56-kserve-6c554c8d5f-9wqvs"]
Sep 11 14:28:29.674267 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:29.674235    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:28:30.503298 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:30.503265    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca73d7cf-268f-46c2-a3a0-3c1c1552a725" path="/var/lib/kubelet/pods/ca73d7cf-268f-46c2-a3a0-3c1c1552a725/volumes"
Sep 11 14:28:41.850839 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:41.850800    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:43.047730 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:43.047695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:43.330154 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:43.330082    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:43.608153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:43.608082    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:44.347456 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:44.347427    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:44.597163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:44.597128    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:45.943716 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:45.943684    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:46.576215 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:46.576184    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:46.759879 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:46.759850    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:47.167613 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:47.167585    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:48.078694 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:48.078652    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:49.054471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.054414    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:49.999343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999305    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"]
Sep 11 14:28:49.999722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999704    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451897d1-e0d0-4146-85d7-e8f410f17c71" containerName="main"
Sep 11 14:28:49.999722 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999721    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="451897d1-e0d0-4146-85d7-e8f410f17c71" containerName="main"
Sep 11 14:28:49.999860 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999739    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca73d7cf-268f-46c2-a3a0-3c1c1552a725" containerName="main"
Sep 11 14:28:49.999860 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999746    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca73d7cf-268f-46c2-a3a0-3c1c1552a725" containerName="main"
Sep 11 14:28:49.999860 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999812    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca73d7cf-268f-46c2-a3a0-3c1c1552a725" containerName="main"
Sep 11 14:28:49.999860 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:49.999823    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="451897d1-e0d0-4146-85d7-e8f410f17c71" containerName="main"
Sep 11 14:28:50.002952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.002928    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.005806 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.005785    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktm9c\""
Sep 11 14:28:50.006760 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.006732    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:28:50.006869 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.006753    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:28:50.006869 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.006747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b7d522-serving-cert\""
Sep 11 14:28:50.006956 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.006738    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:28:50.018216 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.018161    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"]
Sep 11 14:28:50.051776 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.051747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:50.146044 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.145994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.146352 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.146063    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.146352 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.146108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.146352 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.146138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zll\" (UniqueName: \"kubernetes.io/projected/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-kube-api-access-m8zll\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.147135 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.147109    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:50.247448 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.247425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.247563 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.247460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.247563 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.247503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zll\" (UniqueName: \"kubernetes.io/projected/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-kube-api-access-m8zll\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.247677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.247630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.247775 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:28:50.247753    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-serving-cert: secret "maas-api-e2e-ipp-b-b7d522-serving-cert" not found
Sep 11 14:28:50.247842 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:28:50.247827    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls podName:93068b32-7fdd-4be8-9951-f0af7a3d4e5c nodeName:}" failed. No retries permitted until 2026-09-11 14:28:50.747807266 +0000 UTC m=+1864.711422222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls") pod "maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" (UID: "93068b32-7fdd-4be8-9951-f0af7a3d4e5c") : secret "maas-api-e2e-ipp-b-b7d522-serving-cert" not found
Sep 11 14:28:50.248208 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.248187    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.249843 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.249796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.256068 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.256045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zll\" (UniqueName: \"kubernetes.io/projected/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-kube-api-access-m8zll\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.391742 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.391710    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:50.700970 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.700946    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:50.752497 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.752474    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.754693 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.754666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.798394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.798345    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:28:50.828165 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.828138    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"]
Sep 11 14:28:50.831560 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.831545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:50.834013 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.833992    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b7d522-dockercfg-8wmc5\""
Sep 11 14:28:50.834293 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.834277    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7d522\""
Sep 11 14:28:50.853345 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.853325    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"]
Sep 11 14:28:50.861264 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.861238    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"]
Sep 11 14:28:50.864437 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.864420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:50.876613 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.876593    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"]
Sep 11 14:28:50.913911 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.913885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:50.954507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.954438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbds\" (UniqueName: \"kubernetes.io/projected/a217425a-8c2c-4b3d-a315-52f4d04192a4-kube-api-access-lsbds\") pod \"payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:50.954507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.954474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpns\" (UniqueName: \"kubernetes.io/projected/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-kube-api-access-qwpns\") pod \"payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:50.954507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.954502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:50.954776 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:50.954565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a217425a-8c2c-4b3d-a315-52f4d04192a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.055271 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.055243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbds\" (UniqueName: \"kubernetes.io/projected/a217425a-8c2c-4b3d-a315-52f4d04192a4-kube-api-access-lsbds\") pod \"payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.055422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.055283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpns\" (UniqueName: \"kubernetes.io/projected/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-kube-api-access-qwpns\") pod \"payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:51.055422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.055328    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:51.055422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.055361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a217425a-8c2c-4b3d-a315-52f4d04192a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.056099 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.056081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:51.056099 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.056093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a217425a-8c2c-4b3d-a315-52f4d04192a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.063849 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.063822    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpns\" (UniqueName: \"kubernetes.io/projected/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-kube-api-access-qwpns\") pod \"payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:51.063944 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.063893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbds\" (UniqueName: \"kubernetes.io/projected/a217425a-8c2c-4b3d-a315-52f4d04192a4-kube-api-access-lsbds\") pod \"payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.141934 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.141910    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:51.172996 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.172977    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.185512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.184223    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"]
Sep 11 14:28:51.188326 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:28:51.188171    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93068b32_7fdd_4be8_9951_f0af7a3d4e5c.slice/crio-c46df9893569f78e0848d5988066e578d2154321a8320c848c1bd3fc40aafc87 WatchSource:0}: Error finding container c46df9893569f78e0848d5988066e578d2154321a8320c848c1bd3fc40aafc87: Status 404 returned error can't find the container with id c46df9893569f78e0848d5988066e578d2154321a8320c848c1bd3fc40aafc87
Sep 11 14:28:51.417104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.417077    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"]
Sep 11 14:28:51.418532 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:28:51.418505    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4c3ed8_e2ff_4118_99dd_a567e3d2dec1.slice/crio-2775c375ee1666b3e648b4a6cdd3b5673e8ffae7f7d408d7b1ded083e8e2ff3e WatchSource:0}: Error finding container 2775c375ee1666b3e648b4a6cdd3b5673e8ffae7f7d408d7b1ded083e8e2ff3e: Status 404 returned error can't find the container with id 2775c375ee1666b3e648b4a6cdd3b5673e8ffae7f7d408d7b1ded083e8e2ff3e
Sep 11 14:28:51.455378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.455360    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"]
Sep 11 14:28:51.457558 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:28:51.457535    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda217425a_8c2c_4b3d_a315_52f4d04192a4.slice/crio-13f824a9bc5f0a679d729f18f2879521ada81f50d9f97285ae1ab27561d68c62 WatchSource:0}: Error finding container 13f824a9bc5f0a679d729f18f2879521ada81f50d9f97285ae1ab27561d68c62: Status 404 returned error can't find the container with id 13f824a9bc5f0a679d729f18f2879521ada81f50d9f97285ae1ab27561d68c62
Sep 11 14:28:51.936251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.936163    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" event={"ID":"a217425a-8c2c-4b3d-a315-52f4d04192a4","Type":"ContainerStarted","Data":"b932450fb6f6719e632ab759d3688e4d6023f78d12aa0add3cacb9634e316b41"}
Sep 11 14:28:51.936251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.936214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" event={"ID":"a217425a-8c2c-4b3d-a315-52f4d04192a4","Type":"ContainerStarted","Data":"13f824a9bc5f0a679d729f18f2879521ada81f50d9f97285ae1ab27561d68c62"}
Sep 11 14:28:51.936468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.936250    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:28:51.937682 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.937653    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" event={"ID":"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1","Type":"ContainerStarted","Data":"cc06fc94cc458744bc33b1ecb98c18d794320d60730b67f5ef9cd0dac6f42022"}
Sep 11 14:28:51.937807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.937708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" event={"ID":"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1","Type":"ContainerStarted","Data":"2775c375ee1666b3e648b4a6cdd3b5673e8ffae7f7d408d7b1ded083e8e2ff3e"}
Sep 11 14:28:51.937807 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.937743    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:28:51.939196 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.939172    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" event={"ID":"93068b32-7fdd-4be8-9951-f0af7a3d4e5c","Type":"ContainerStarted","Data":"74dd1caf39f55ba232729bfd8e206f8e3e83c08dabe812679e31deca8bfd0e55"}
Sep 11 14:28:51.939291 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.939217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" event={"ID":"93068b32-7fdd-4be8-9951-f0af7a3d4e5c","Type":"ContainerStarted","Data":"c46df9893569f78e0848d5988066e578d2154321a8320c848c1bd3fc40aafc87"}
Sep 11 14:28:51.939291 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.939257    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:28:51.956683 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.956644    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" podStartSLOduration=1.956630651 podStartE2EDuration="1.956630651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:51.955097736 +0000 UTC m=+1865.918712712" watchObservedRunningTime="2026-09-11 14:28:51.956630651 +0000 UTC m=+1865.920245614"
Sep 11 14:28:51.974780 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.974741    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" podStartSLOduration=1.974729175 podStartE2EDuration="1.974729175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:51.972543827 +0000 UTC m=+1865.936158826" watchObservedRunningTime="2026-09-11 14:28:51.974729175 +0000 UTC m=+1865.938344147"
Sep 11 14:28:51.990440 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:51.990396    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" podStartSLOduration=2.6918680310000003 podStartE2EDuration="2.990380602s" totalImagesPullingTime="298.512571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:51.989755031 +0000 UTC m=+1865.953369999" watchObservedRunningTime="2026-09-11 14:28:51.990380602 +0000 UTC m=+1865.953995581"
Sep 11 14:28:54.768714 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:54.768670    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:28:57.949975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:28:57.949944    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:29:02.946615 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:02.946583    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:29:02.947083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:02.946776    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:29:03.477234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:03.477199    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:12.564123 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:12.563999    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:13.307171 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:13.307133    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:14.137204 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:14.137132    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:14.898087 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:14.898019    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:15.675436 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:15.675404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:16.712052 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:16.712003    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:22.105839 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:22.105791    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc_1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1/payload-processing/0.log"
Sep 11 14:29:24.478727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.478696    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"]
Sep 11 14:29:24.482260 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.482242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.487306 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.487285    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-30989f\"/\"kube-root-ca.crt\""
Sep 11 14:29:24.487428 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.487285    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-30989f\"/\"ipp-route-30989f-kserve-self-signed-certs\""
Sep 11 14:29:24.487428 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.487285    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-30989f\"/\"default-dockercfg-tz4ft\""
Sep 11 14:29:24.495489 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.495472    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-30989f\"/\"openshift-service-ca.crt\""
Sep 11 14:29:24.500645 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.500624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gnrp\" (UniqueName: \"kubernetes.io/projected/2b9d2057-e36a-4677-878f-9e6b24bed1f0-kube-api-access-4gnrp\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.500735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.500653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tls-certs\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.500735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.500705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-home\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.500847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.500735    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-model-cache\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.500847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.500756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tmp-dir\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.500847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.500802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-dshm\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.503480 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.503461    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"]
Sep 11 14:29:24.542532 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.542506    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:24.601698 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.601678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-model-cache\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.601782 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.601702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tmp-dir\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.601782 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.601721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-dshm\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.601891 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.601786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gnrp\" (UniqueName: \"kubernetes.io/projected/2b9d2057-e36a-4677-878f-9e6b24bed1f0-kube-api-access-4gnrp\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.601891 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.601804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tls-certs\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.601891 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.601837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-home\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.602160 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.602131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tmp-dir\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.602250 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.602181    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-home\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.602250 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.602215    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-model-cache\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.603975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.603959    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-dshm\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.604163 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.604147    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tls-certs\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.612130 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.612108    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gnrp\" (UniqueName: \"kubernetes.io/projected/2b9d2057-e36a-4677-878f-9e6b24bed1f0-kube-api-access-4gnrp\") pod \"ipp-route-30989f-kserve-79b7464f55-f5t8w\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") " pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:24.790822 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:24.790763    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:25.064238 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:25.064166    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"]
Sep 11 14:29:25.067263 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:29:25.067233    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b9d2057_e36a_4677_878f_9e6b24bed1f0.slice/crio-7775ea9cda2aa583af7495c7b3bbb6518f84dfb205838144eb1bc17dc4c7a896 WatchSource:0}: Error finding container 7775ea9cda2aa583af7495c7b3bbb6518f84dfb205838144eb1bc17dc4c7a896: Status 404 returned error can't find the container with id 7775ea9cda2aa583af7495c7b3bbb6518f84dfb205838144eb1bc17dc4c7a896
Sep 11 14:29:25.098342 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:25.098320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w" event={"ID":"2b9d2057-e36a-4677-878f-9e6b24bed1f0","Type":"ContainerStarted","Data":"7775ea9cda2aa583af7495c7b3bbb6518f84dfb205838144eb1bc17dc4c7a896"}
Sep 11 14:29:26.104294 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:26.104256    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w" event={"ID":"2b9d2057-e36a-4677-878f-9e6b24bed1f0","Type":"ContainerStarted","Data":"6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb"}
Sep 11 14:29:26.131917 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:26.131863    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w" podStartSLOduration=2.13184913 podStartE2EDuration="2.13184913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:26.131192544 +0000 UTC m=+1900.094807518" watchObservedRunningTime="2026-09-11 14:29:26.13184913 +0000 UTC m=+1900.095464101"
Sep 11 14:29:28.692844 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:28.692815    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:29:34.791190 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:34.791157    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:34.791549 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:34.791233    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:34.803103 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:34.803081    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:35.154875 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:35.154808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:29:51.771341 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:51.771309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:29:58.312524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:58.312479    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc_1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1/payload-processing/0.log"
Sep 11 14:29:58.643274 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:58.643202    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:59.212969 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:59.212935    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:29:59.559243 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:29:59.559164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:00.202632 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.202600    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"]
Sep 11 14:30:00.206353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.206336    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"
Sep 11 14:30:00.210356 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.210335    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\""
Sep 11 14:30:00.215348 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.215332    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"
Sep 11 14:30:00.223473 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.223441    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"]
Sep 11 14:30:00.227817 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.227794    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"
Sep 11 14:30:00.230383 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.230357    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"]
Sep 11 14:30:00.239562 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.239543    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"]
Sep 11 14:30:00.255941 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.255918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"
Sep 11 14:30:00.527583 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.527560    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"]
Sep 11 14:30:00.529777 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:30:00.529746    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef98821_7339_4eae_b17c_00fc696247b4.slice/crio-d4e5be1cf837b8a0f22b1eef4d416b18ccbddaa99ccbfa8f72dfa804a8384bb7 WatchSource:0}: Error finding container d4e5be1cf837b8a0f22b1eef4d416b18ccbddaa99ccbfa8f72dfa804a8384bb7: Status 404 returned error can't find the container with id d4e5be1cf837b8a0f22b1eef4d416b18ccbddaa99ccbfa8f72dfa804a8384bb7
Sep 11 14:30:00.531601 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.531579    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:30:00.560576 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:30:00.560552    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87af1fdb_8d99_4198_8a02_7f2135aaed1d.slice/crio-b1b91b5fce52d068e2cf4085318ef77f7d22b57260978c36c7b7337e702e7d9c WatchSource:0}: Error finding container b1b91b5fce52d068e2cf4085318ef77f7d22b57260978c36c7b7337e702e7d9c: Status 404 returned error can't find the container with id b1b91b5fce52d068e2cf4085318ef77f7d22b57260978c36c7b7337e702e7d9c
Sep 11 14:30:00.560950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:00.560576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"]
Sep 11 14:30:01.260842 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:01.260796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2" event={"ID":"1ef98821-7339-4eae-b17c-00fc696247b4","Type":"ContainerStarted","Data":"d4e5be1cf837b8a0f22b1eef4d416b18ccbddaa99ccbfa8f72dfa804a8384bb7"}
Sep 11 14:30:01.261905 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:01.261879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8" event={"ID":"87af1fdb-8d99-4198-8a02-7f2135aaed1d","Type":"ContainerStarted","Data":"b1b91b5fce52d068e2cf4085318ef77f7d22b57260978c36c7b7337e702e7d9c"}
Sep 11 14:30:01.502909 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:01.502878    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:02.432378 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:02.432338    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:02.617744 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:02.617711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:03.125531 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:03.121059    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:04.128727 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:04.128686    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:04.280558 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:04.280533    2558 generic.go:356] "Generic (PLEG): container finished" podID="1ef98821-7339-4eae-b17c-00fc696247b4" containerID="c81e5f66fd6d97c2907a28a6df1a0ebd2167216f217e20867c4a52bcecb6b1f3" exitCode=0
Sep 11 14:30:04.280558 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:04.280567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2" event={"ID":"1ef98821-7339-4eae-b17c-00fc696247b4","Type":"ContainerDied","Data":"c81e5f66fd6d97c2907a28a6df1a0ebd2167216f217e20867c4a52bcecb6b1f3"}
Sep 11 14:30:05.107198 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:05.107167    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:05.401972 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:05.401949    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"
Sep 11 14:30:06.292079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:06.292044    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2" event={"ID":"1ef98821-7339-4eae-b17c-00fc696247b4","Type":"ContainerDied","Data":"d4e5be1cf837b8a0f22b1eef4d416b18ccbddaa99ccbfa8f72dfa804a8384bb7"}
Sep 11 14:30:06.292079 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:06.292076    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"
Sep 11 14:30:06.292283 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:06.292099    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e5be1cf837b8a0f22b1eef4d416b18ccbddaa99ccbfa8f72dfa804a8384bb7"
Sep 11 14:30:06.824904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:06.824870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:07.757133 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:07.757103    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:08.116632 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:08.116548    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:08.229720 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:08.229687    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:08.489492 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:08.489462    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:08.595761 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:08.595727    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:13.113344 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:13.113309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:30:15.184083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:15.184048    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:18.353882 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:18.353801    2558 generic.go:356] "Generic (PLEG): container finished" podID="87af1fdb-8d99-4198-8a02-7f2135aaed1d" containerID="1584464ba15ae0e93db76f21dba7553afc1bd9b78009e27931b6f50ced6a00d0" exitCode=0
Sep 11 14:30:18.353882 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:18.353845    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8" event={"ID":"87af1fdb-8d99-4198-8a02-7f2135aaed1d","Type":"ContainerDied","Data":"1584464ba15ae0e93db76f21dba7553afc1bd9b78009e27931b6f50ced6a00d0"}
Sep 11 14:30:19.485065 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:19.485018    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"
Sep 11 14:30:20.364101 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:20.364070    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8"
Sep 11 14:30:20.364101 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:20.364090    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jdpk8" event={"ID":"87af1fdb-8d99-4198-8a02-7f2135aaed1d","Type":"ContainerDied","Data":"b1b91b5fce52d068e2cf4085318ef77f7d22b57260978c36c7b7337e702e7d9c"}
Sep 11 14:30:20.364373 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:20.364134    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1b91b5fce52d068e2cf4085318ef77f7d22b57260978c36c7b7337e702e7d9c"
Sep 11 14:30:21.765774 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:21.765744    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:29.167726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:29.167690    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:31.079170 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:31.079134    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:32.193197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:32.193162    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:32.365148 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:32.365121    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:33.472256 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:33.472223    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:34.521040 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:34.520981    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:35.346173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:35.346143    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:35.446081 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:35.446048    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:36.298389 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:36.298355    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:37.200685 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:37.200637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:38.328867 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:38.328832    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:39.279266 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:39.279183    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:43.890479 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:43.890442    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:44.741376 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:44.741344    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:44.853175 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:44.853141    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:44.958344 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:44.958298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:45.076921 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:45.076838    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:45.199514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:45.199478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:45.464935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:45.464896    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:45.623320 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:45.623290    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:46.468257 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:46.468216    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:47.670439 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:47.670402    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:48.529634 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:48.529595    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:49.188347 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:49.188317    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:30:49.363512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:49.363472    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:51.880323 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:51.880285    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:52.110359 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:52.110325    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:52.337874 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:52.337842    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:56.287945 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:56.287915    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:57.328576 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:57.328542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:58.502806 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:58.502771    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:30:59.557258 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:30:59.557226    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:00.463234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:00.463206    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:01.781359 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:01.781326    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:09.794935 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:09.794900    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:31:11.990046 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:11.990000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:12.103680 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:12.103651    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:12.249430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:12.249342    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:12.397726 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:12.397698    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:12.773487 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:12.773453    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:12.872317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:12.872283    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:14.040077 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:14.040044    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:15.259321 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:15.259288    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:16.259050 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:16.258995    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:17.730552 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:17.730518    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"]
Sep 11 14:31:17.826019 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:17.825981    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:22.630658 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:22.630620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:31:32.749104 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:32.749063    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w" podUID="2b9d2057-e36a-4677-878f-9e6b24bed1f0" containerName="main" containerID="cri-o://6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb" gracePeriod=45
Sep 11 14:31:32.995372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:32.995350    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:31:33.089976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.089893    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-dshm\") pod \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") "
Sep 11 14:31:33.089976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.089945    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tmp-dir\") pod \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") "
Sep 11 14:31:33.089976 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.089964    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-model-cache\") pod \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") "
Sep 11 14:31:33.090259 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.089997    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tls-certs\") pod \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") "
Sep 11 14:31:33.090259 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090046    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gnrp\" (UniqueName: \"kubernetes.io/projected/2b9d2057-e36a-4677-878f-9e6b24bed1f0-kube-api-access-4gnrp\") pod \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") "
Sep 11 14:31:33.090259 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090071    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-home\") pod \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\" (UID: \"2b9d2057-e36a-4677-878f-9e6b24bed1f0\") "
Sep 11 14:31:33.090259 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090215    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b9d2057-e36a-4677-878f-9e6b24bed1f0" (UID: "2b9d2057-e36a-4677-878f-9e6b24bed1f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:33.090467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090283    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b9d2057-e36a-4677-878f-9e6b24bed1f0" (UID: "2b9d2057-e36a-4677-878f-9e6b24bed1f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:33.090467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090387    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:31:33.090467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090402    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:31:33.090467 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.090411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-home" (OuterVolumeSpecName: "home") pod "2b9d2057-e36a-4677-878f-9e6b24bed1f0" (UID: "2b9d2057-e36a-4677-878f-9e6b24bed1f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:33.092194 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.092168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-dshm" (OuterVolumeSpecName: "dshm") pod "2b9d2057-e36a-4677-878f-9e6b24bed1f0" (UID: "2b9d2057-e36a-4677-878f-9e6b24bed1f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:33.092194 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.092166    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9d2057-e36a-4677-878f-9e6b24bed1f0-kube-api-access-4gnrp" (OuterVolumeSpecName: "kube-api-access-4gnrp") pod "2b9d2057-e36a-4677-878f-9e6b24bed1f0" (UID: "2b9d2057-e36a-4677-878f-9e6b24bed1f0"). InnerVolumeSpecName "kube-api-access-4gnrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:31:33.092351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.092234    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b9d2057-e36a-4677-878f-9e6b24bed1f0" (UID: "2b9d2057-e36a-4677-878f-9e6b24bed1f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:31:33.191408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.191375    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d2057-e36a-4677-878f-9e6b24bed1f0-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:31:33.191408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.191402    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gnrp\" (UniqueName: \"kubernetes.io/projected/2b9d2057-e36a-4677-878f-9e6b24bed1f0-kube-api-access-4gnrp\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:31:33.191408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.191411    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:31:33.191600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.191421    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d2057-e36a-4677-878f-9e6b24bed1f0-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:31:33.698630 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.698595    2558 generic.go:356] "Generic (PLEG): container finished" podID="2b9d2057-e36a-4677-878f-9e6b24bed1f0" containerID="6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb" exitCode=0
Sep 11 14:31:33.698816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.698640    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"
Sep 11 14:31:33.698816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.698682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w" event={"ID":"2b9d2057-e36a-4677-878f-9e6b24bed1f0","Type":"ContainerDied","Data":"6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb"}
Sep 11 14:31:33.698816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.698724    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w" event={"ID":"2b9d2057-e36a-4677-878f-9e6b24bed1f0","Type":"ContainerDied","Data":"7775ea9cda2aa583af7495c7b3bbb6518f84dfb205838144eb1bc17dc4c7a896"}
Sep 11 14:31:33.698816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.698742    2558 scope.go:117] "RemoveContainer" containerID="6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb"
Sep 11 14:31:33.707709 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.707693    2558 scope.go:117] "RemoveContainer" containerID="6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb"
Sep 11 14:31:33.707992 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:31:33.707975    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb\": container with ID starting with 6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb not found: ID does not exist" containerID="6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb"
Sep 11 14:31:33.708067 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.708000    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb"} err="failed to get container status \"6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb\": rpc error: code = NotFound desc = could not find container \"6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb\": container with ID starting with 6f7872f1bfe1854f7ce64d6c725d1b53abcd6e6d785e1f03011bfd8559f6fceb not found: ID does not exist"
Sep 11 14:31:33.724967 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.724945    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"]
Sep 11 14:31:33.726664 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:33.726643    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-30989f/ipp-route-30989f-kserve-79b7464f55-f5t8w"]
Sep 11 14:31:34.504084 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:34.504024    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9d2057-e36a-4677-878f-9e6b24bed1f0" path="/var/lib/kubelet/pods/2b9d2057-e36a-4677-878f-9e6b24bed1f0/volumes"
Sep 11 14:31:53.901083 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:53.901041    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:31:55.656838 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:31:55.656800    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:01.329874 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:01.329816    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:01.431546 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:01.431510    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:01.641840 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:01.641756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:02.232599 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:02.232563    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:03.253905 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:03.253864    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:04.158535 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:04.158496    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:04.719413 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:04.719381    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:05.139752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:05.139649    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:05.998979 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:05.998938    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:12.649964 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.649892    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"]
Sep 11 14:32:12.650367 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.650138    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" podUID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" containerName="maas-api" containerID="cri-o://74dd1caf39f55ba232729bfd8e206f8e3e83c08dabe812679e31deca8bfd0e55" gracePeriod=30
Sep 11 14:32:12.689080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.689051    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:12.696647 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.696619    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"]
Sep 11 14:32:12.702317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.702300    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b7d522-29818950-t24d2"]
Sep 11 14:32:12.732649 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.732621    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"]
Sep 11 14:32:12.732863 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.732840    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" podUID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" containerName="payload-processing" containerID="cri-o://cc06fc94cc458744bc33b1ecb98c18d794320d60730b67f5ef9cd0dac6f42022" gracePeriod=30
Sep 11 14:32:12.741897 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.741872    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"]
Sep 11 14:32:12.742121 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.742099    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" podUID="a217425a-8c2c-4b3d-a315-52f4d04192a4" containerName="payload-pre-processing" containerID="cri-o://b932450fb6f6719e632ab759d3688e4d6023f78d12aa0add3cacb9634e316b41" gracePeriod=30
Sep 11 14:32:12.791768 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.791743    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:12.891372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.891333    2558 generic.go:356] "Generic (PLEG): container finished" podID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" containerID="cc06fc94cc458744bc33b1ecb98c18d794320d60730b67f5ef9cd0dac6f42022" exitCode=0
Sep 11 14:32:12.891539 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.891408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" event={"ID":"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1","Type":"ContainerDied","Data":"cc06fc94cc458744bc33b1ecb98c18d794320d60730b67f5ef9cd0dac6f42022"}
Sep 11 14:32:12.893493 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.893279    2558 generic.go:356] "Generic (PLEG): container finished" podID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" containerID="74dd1caf39f55ba232729bfd8e206f8e3e83c08dabe812679e31deca8bfd0e55" exitCode=0
Sep 11 14:32:12.893493 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.893438    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" event={"ID":"93068b32-7fdd-4be8-9951-f0af7a3d4e5c","Type":"ContainerDied","Data":"74dd1caf39f55ba232729bfd8e206f8e3e83c08dabe812679e31deca8bfd0e55"}
Sep 11 14:32:12.894809 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.894776    2558 generic.go:356] "Generic (PLEG): container finished" podID="a217425a-8c2c-4b3d-a315-52f4d04192a4" containerID="b932450fb6f6719e632ab759d3688e4d6023f78d12aa0add3cacb9634e316b41" exitCode=0
Sep 11 14:32:12.894928 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.894828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" event={"ID":"a217425a-8c2c-4b3d-a315-52f4d04192a4","Type":"ContainerDied","Data":"b932450fb6f6719e632ab759d3688e4d6023f78d12aa0add3cacb9634e316b41"}
Sep 11 14:32:12.898485 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.898456    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:12.945020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.944979    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" podUID="a217425a-8c2c-4b3d-a315-52f4d04192a4" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.74:9004: connect: connection refused"
Sep 11 14:32:12.988902 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:12.988874    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:32:13.001309 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.001281    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:13.019568 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.019543    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:32:13.028422 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.028401    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:32:13.097095 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.097063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:13.131014 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.130982    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls\") pod \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") "
Sep 11 14:32:13.131173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131024    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-metrics-certs\") pod \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") "
Sep 11 14:32:13.131173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131069    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a217425a-8c2c-4b3d-a315-52f4d04192a4-plugins-config-volume\") pod \"a217425a-8c2c-4b3d-a315-52f4d04192a4\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") "
Sep 11 14:32:13.131173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-openshift-service-ca\") pod \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") "
Sep 11 14:32:13.131173 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131161    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8zll\" (UniqueName: \"kubernetes.io/projected/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-kube-api-access-m8zll\") pod \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\" (UID: \"93068b32-7fdd-4be8-9951-f0af7a3d4e5c\") "
Sep 11 14:32:13.131343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131228    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-plugins-config-volume\") pod \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") "
Sep 11 14:32:13.131343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131269    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsbds\" (UniqueName: \"kubernetes.io/projected/a217425a-8c2c-4b3d-a315-52f4d04192a4-kube-api-access-lsbds\") pod \"a217425a-8c2c-4b3d-a315-52f4d04192a4\" (UID: \"a217425a-8c2c-4b3d-a315-52f4d04192a4\") "
Sep 11 14:32:13.131343 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwpns\" (UniqueName: \"kubernetes.io/projected/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-kube-api-access-qwpns\") pod \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\" (UID: \"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1\") "
Sep 11 14:32:13.131715 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93068b32-7fdd-4be8-9951-f0af7a3d4e5c" (UID: "93068b32-7fdd-4be8-9951-f0af7a3d4e5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:13.131715 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131665    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" (UID: "1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:13.131866 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.131746    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a217425a-8c2c-4b3d-a315-52f4d04192a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a217425a-8c2c-4b3d-a315-52f4d04192a4" (UID: "a217425a-8c2c-4b3d-a315-52f4d04192a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:13.133202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.133177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93068b32-7fdd-4be8-9951-f0af7a3d4e5c" (UID: "93068b32-7fdd-4be8-9951-f0af7a3d4e5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:32:13.133348 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.133326    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a217425a-8c2c-4b3d-a315-52f4d04192a4-kube-api-access-lsbds" (OuterVolumeSpecName: "kube-api-access-lsbds") pod "a217425a-8c2c-4b3d-a315-52f4d04192a4" (UID: "a217425a-8c2c-4b3d-a315-52f4d04192a4"). InnerVolumeSpecName "kube-api-access-lsbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:13.133470 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.133451    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-kube-api-access-qwpns" (OuterVolumeSpecName: "kube-api-access-qwpns") pod "1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" (UID: "1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1"). InnerVolumeSpecName "kube-api-access-qwpns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:13.133629 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.133605    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93068b32-7fdd-4be8-9951-f0af7a3d4e5c" (UID: "93068b32-7fdd-4be8-9951-f0af7a3d4e5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:32:13.133629 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.133616    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-kube-api-access-m8zll" (OuterVolumeSpecName: "kube-api-access-m8zll") pod "93068b32-7fdd-4be8-9951-f0af7a3d4e5c" (UID: "93068b32-7fdd-4be8-9951-f0af7a3d4e5c"). InnerVolumeSpecName "kube-api-access-m8zll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:13.232226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232195    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsbds\" (UniqueName: \"kubernetes.io/projected/a217425a-8c2c-4b3d-a315-52f4d04192a4-kube-api-access-lsbds\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232224    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpns\" (UniqueName: \"kubernetes.io/projected/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-kube-api-access-qwpns\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232234    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232242    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232250    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a217425a-8c2c-4b3d-a315-52f4d04192a4-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232258    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232265    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zll\" (UniqueName: \"kubernetes.io/projected/93068b32-7fdd-4be8-9951-f0af7a3d4e5c-kube-api-access-m8zll\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.232398 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.232275    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:13.465655 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.465624    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:13.572003 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.571936    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:13.901223 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.901141    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"
Sep 11 14:32:13.901223 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.901173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk" event={"ID":"a217425a-8c2c-4b3d-a315-52f4d04192a4","Type":"ContainerDied","Data":"13f824a9bc5f0a679d729f18f2879521ada81f50d9f97285ae1ab27561d68c62"}
Sep 11 14:32:13.901701 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.901246    2558 scope.go:117] "RemoveContainer" containerID="b932450fb6f6719e632ab759d3688e4d6023f78d12aa0add3cacb9634e316b41"
Sep 11 14:32:13.902595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.902520    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"
Sep 11 14:32:13.902595 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.902532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" event={"ID":"1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1","Type":"ContainerDied","Data":"2775c375ee1666b3e648b4a6cdd3b5673e8ffae7f7d408d7b1ded083e8e2ff3e"}
Sep 11 14:32:13.904224 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.904203    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"
Sep 11 14:32:13.904224 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.904214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" event={"ID":"93068b32-7fdd-4be8-9951-f0af7a3d4e5c","Type":"ContainerDied","Data":"c46df9893569f78e0848d5988066e578d2154321a8320c848c1bd3fc40aafc87"}
Sep 11 14:32:13.911213 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.911194    2558 scope.go:117] "RemoveContainer" containerID="cc06fc94cc458744bc33b1ecb98c18d794320d60730b67f5ef9cd0dac6f42022"
Sep 11 14:32:13.922110 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.922088    2558 scope.go:117] "RemoveContainer" containerID="74dd1caf39f55ba232729bfd8e206f8e3e83c08dabe812679e31deca8bfd0e55"
Sep 11 14:32:13.945565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.945542    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc" podUID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.73:9004: i/o timeout"
Sep 11 14:32:13.947981 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.947962    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"]
Sep 11 14:32:13.955783 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.955763    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7d522-7968dd9999-454gc"]
Sep 11 14:32:13.978342 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.978324    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"]
Sep 11 14:32:13.981538 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:13.981519    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx"]
Sep 11 14:32:14.013869 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.013849    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"]
Sep 11 14:32:14.018540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.018522    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7d522-5c7b8c8d74-jxkrk"]
Sep 11 14:32:14.502944 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.502911    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef98821-7339-4eae-b17c-00fc696247b4" path="/var/lib/kubelet/pods/1ef98821-7339-4eae-b17c-00fc696247b4/volumes"
Sep 11 14:32:14.503272 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.503259    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" path="/var/lib/kubelet/pods/1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1/volumes"
Sep 11 14:32:14.503600 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.503588    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" path="/var/lib/kubelet/pods/93068b32-7fdd-4be8-9951-f0af7a3d4e5c/volumes"
Sep 11 14:32:14.503891 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.503881    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a217425a-8c2c-4b3d-a315-52f4d04192a4" path="/var/lib/kubelet/pods/a217425a-8c2c-4b3d-a315-52f4d04192a4/volumes"
Sep 11 14:32:14.697399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:14.697366    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:15.945847 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:15.945793    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7d522-75d5d47856-5rblx" podUID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.72:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 14:32:16.056220 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:16.056188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:16.208544 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:16.208457    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:32:19.879466 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:19.879435    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:19.987202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:19.987159    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:20.227628 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.227594    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:20.585317 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585238    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"]
Sep 11 14:32:20.585648 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585631    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" containerName="payload-processing"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585650    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" containerName="payload-processing"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585663    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef98821-7339-4eae-b17c-00fc696247b4" containerName="cleanup"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585670    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef98821-7339-4eae-b17c-00fc696247b4" containerName="cleanup"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585681    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a217425a-8c2c-4b3d-a315-52f4d04192a4" containerName="payload-pre-processing"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585689    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a217425a-8c2c-4b3d-a315-52f4d04192a4" containerName="payload-pre-processing"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585697    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87af1fdb-8d99-4198-8a02-7f2135aaed1d" containerName="cleanup"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585706    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="87af1fdb-8d99-4198-8a02-7f2135aaed1d" containerName="cleanup"
Sep 11 14:32:20.585735 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585735    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9d2057-e36a-4677-878f-9e6b24bed1f0" containerName="main"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585743    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9d2057-e36a-4677-878f-9e6b24bed1f0" containerName="main"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585762    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" containerName="maas-api"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585770    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" containerName="maas-api"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585860    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a217425a-8c2c-4b3d-a315-52f4d04192a4" containerName="payload-pre-processing"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585873    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9d2057-e36a-4677-878f-9e6b24bed1f0" containerName="main"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585884    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4c3ed8-e2ff-4118-99dd-a567e3d2dec1" containerName="payload-processing"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585894    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="93068b32-7fdd-4be8-9951-f0af7a3d4e5c" containerName="maas-api"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585902    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef98821-7339-4eae-b17c-00fc696247b4" containerName="cleanup"
Sep 11 14:32:20.586161 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.585914    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="87af1fdb-8d99-4198-8a02-7f2135aaed1d" containerName="cleanup"
Sep 11 14:32:20.590788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.590768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:20.593468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.593446    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:32:20.593573 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.593555    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\""
Sep 11 14:32:20.599555 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.599533    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"]
Sep 11 14:32:20.697799 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.697770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d591fbf9-3720-4475-bb45-2387c8abf465-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55\" (UID: \"d591fbf9-3720-4475-bb45-2387c8abf465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:20.799075 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.799008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d591fbf9-3720-4475-bb45-2387c8abf465-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55\" (UID: \"d591fbf9-3720-4475-bb45-2387c8abf465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:20.799602 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.799583    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d591fbf9-3720-4475-bb45-2387c8abf465-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55\" (UID: \"d591fbf9-3720-4475-bb45-2387c8abf465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:20.901206 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:20.901132    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:21.177203 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:21.177121    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"]
Sep 11 14:32:21.179939 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:32:21.179912    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd591fbf9_3720_4475_bb45_2387c8abf465.slice/crio-896a0664a4f1f18b0953062b12e86ff97b4ad55bb36ab9a1cbe9d5ca832e0cdf WatchSource:0}: Error finding container 896a0664a4f1f18b0953062b12e86ff97b4ad55bb36ab9a1cbe9d5ca832e0cdf: Status 404 returned error can't find the container with id 896a0664a4f1f18b0953062b12e86ff97b4ad55bb36ab9a1cbe9d5ca832e0cdf
Sep 11 14:32:21.638801 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:21.638752    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:21.945558 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:21.945521    2558 generic.go:356] "Generic (PLEG): container finished" podID="d591fbf9-3720-4475-bb45-2387c8abf465" containerID="d0b7c20503621aa2a390de8c42827e48476d71dd144052c643c31f9cae568798" exitCode=0
Sep 11 14:32:21.945953 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:21.945605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55" event={"ID":"d591fbf9-3720-4475-bb45-2387c8abf465","Type":"ContainerDied","Data":"d0b7c20503621aa2a390de8c42827e48476d71dd144052c643c31f9cae568798"}
Sep 11 14:32:21.945953 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:21.945674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55" event={"ID":"d591fbf9-3720-4475-bb45-2387c8abf465","Type":"ContainerStarted","Data":"896a0664a4f1f18b0953062b12e86ff97b4ad55bb36ab9a1cbe9d5ca832e0cdf"}
Sep 11 14:32:22.568374 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:22.568337    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:23.069872 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.069847    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:23.221009 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.220948    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d591fbf9-3720-4475-bb45-2387c8abf465-maas-api-service-ca\") pod \"d591fbf9-3720-4475-bb45-2387c8abf465\" (UID: \"d591fbf9-3720-4475-bb45-2387c8abf465\") "
Sep 11 14:32:23.221325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.221301    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d591fbf9-3720-4475-bb45-2387c8abf465-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d591fbf9-3720-4475-bb45-2387c8abf465" (UID: "d591fbf9-3720-4475-bb45-2387c8abf465"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:23.322146 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.322118    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d591fbf9-3720-4475-bb45-2387c8abf465-maas-api-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:23.608433 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.608359    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:23.955406 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.955376    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"
Sep 11 14:32:23.955581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.955411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55" event={"ID":"d591fbf9-3720-4475-bb45-2387c8abf465","Type":"ContainerDied","Data":"896a0664a4f1f18b0953062b12e86ff97b4ad55bb36ab9a1cbe9d5ca832e0cdf"}
Sep 11 14:32:23.955581 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:23.955467    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="896a0664a4f1f18b0953062b12e86ff97b4ad55bb36ab9a1cbe9d5ca832e0cdf"
Sep 11 14:32:31.424927 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.424890    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:31.475112 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.475084    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"]
Sep 11 14:32:31.475325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.475303    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7" podUID="8ca15502-7c36-491c-afab-96654299f89d" containerName="payload-processing" containerID="cri-o://fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64" gracePeriod=30
Sep 11 14:32:31.484806 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.484777    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"]
Sep 11 14:32:31.484981 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.484963    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c" podUID="a186b216-4532-44a4-8f51-56ca6ccf4439" containerName="payload-pre-processing" containerID="cri-o://8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523" gracePeriod=30
Sep 11 14:32:31.549195 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.549159    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:31.665010 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.664982    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:31.745251 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.745221    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:32:31.747799 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.747772    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:32:31.776563 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.776535    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:31.833982 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.833954    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"]
Sep 11 14:32:31.838854 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.838827    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1e2e7d-qbt55"]
Sep 11 14:32:31.888830 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.888807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f5rw\" (UniqueName: \"kubernetes.io/projected/8ca15502-7c36-491c-afab-96654299f89d-kube-api-access-4f5rw\") pod \"8ca15502-7c36-491c-afab-96654299f89d\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") "
Sep 11 14:32:31.888933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.888857    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a186b216-4532-44a4-8f51-56ca6ccf4439-plugins-config-volume\") pod \"a186b216-4532-44a4-8f51-56ca6ccf4439\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") "
Sep 11 14:32:31.888933 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.888879    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-989qf\" (UniqueName: \"kubernetes.io/projected/a186b216-4532-44a4-8f51-56ca6ccf4439-kube-api-access-989qf\") pod \"a186b216-4532-44a4-8f51-56ca6ccf4439\" (UID: \"a186b216-4532-44a4-8f51-56ca6ccf4439\") "
Sep 11 14:32:31.889060 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.888934    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca15502-7c36-491c-afab-96654299f89d-plugins-config-volume\") pod \"8ca15502-7c36-491c-afab-96654299f89d\" (UID: \"8ca15502-7c36-491c-afab-96654299f89d\") "
Sep 11 14:32:31.889273 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.889245    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a186b216-4532-44a4-8f51-56ca6ccf4439-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a186b216-4532-44a4-8f51-56ca6ccf4439" (UID: "a186b216-4532-44a4-8f51-56ca6ccf4439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:31.889353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.889333    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca15502-7c36-491c-afab-96654299f89d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ca15502-7c36-491c-afab-96654299f89d" (UID: "8ca15502-7c36-491c-afab-96654299f89d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:31.891003 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.890977    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a186b216-4532-44a4-8f51-56ca6ccf4439-kube-api-access-989qf" (OuterVolumeSpecName: "kube-api-access-989qf") pod "a186b216-4532-44a4-8f51-56ca6ccf4439" (UID: "a186b216-4532-44a4-8f51-56ca6ccf4439"). InnerVolumeSpecName "kube-api-access-989qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:31.891172 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.891056    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca15502-7c36-491c-afab-96654299f89d-kube-api-access-4f5rw" (OuterVolumeSpecName: "kube-api-access-4f5rw") pod "8ca15502-7c36-491c-afab-96654299f89d" (UID: "8ca15502-7c36-491c-afab-96654299f89d"). InnerVolumeSpecName "kube-api-access-4f5rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:31.924369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.924343    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:31.989961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.989939    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5rw\" (UniqueName: \"kubernetes.io/projected/8ca15502-7c36-491c-afab-96654299f89d-kube-api-access-4f5rw\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:31.989961 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.989962    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a186b216-4532-44a4-8f51-56ca6ccf4439-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:31.990147 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.989971    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-989qf\" (UniqueName: \"kubernetes.io/projected/a186b216-4532-44a4-8f51-56ca6ccf4439-kube-api-access-989qf\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:31.990147 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.989979    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca15502-7c36-491c-afab-96654299f89d-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:31.992958 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.992936    2558 generic.go:356] "Generic (PLEG): container finished" podID="8ca15502-7c36-491c-afab-96654299f89d" containerID="fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64" exitCode=0
Sep 11 14:32:31.993089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.992985    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"
Sep 11 14:32:31.993089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.993003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7" event={"ID":"8ca15502-7c36-491c-afab-96654299f89d","Type":"ContainerDied","Data":"fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64"}
Sep 11 14:32:31.993089 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.993078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7" event={"ID":"8ca15502-7c36-491c-afab-96654299f89d","Type":"ContainerDied","Data":"56ef09c747f8a67b1e8d0681b9ad82b49764197b13970ebf691c79d7cba6e744"}
Sep 11 14:32:31.993253 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.993111    2558 scope.go:117] "RemoveContainer" containerID="fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64"
Sep 11 14:32:31.994481 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.994425    2558 generic.go:356] "Generic (PLEG): container finished" podID="a186b216-4532-44a4-8f51-56ca6ccf4439" containerID="8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523" exitCode=0
Sep 11 14:32:31.994574 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.994483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c" event={"ID":"a186b216-4532-44a4-8f51-56ca6ccf4439","Type":"ContainerDied","Data":"8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523"}
Sep 11 14:32:31.994574 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.994531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c" event={"ID":"a186b216-4532-44a4-8f51-56ca6ccf4439","Type":"ContainerDied","Data":"4aad243ac8ef9f8e09239c1d4dce14a9ccc0463cadca0440c5728e7f37eeafde"}
Sep 11 14:32:31.994574 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:31.994468    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"
Sep 11 14:32:32.002155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.002133    2558 scope.go:117] "RemoveContainer" containerID="fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64"
Sep 11 14:32:32.002443 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:32:32.002421    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64\": container with ID starting with fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64 not found: ID does not exist" containerID="fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64"
Sep 11 14:32:32.002524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.002449    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64"} err="failed to get container status \"fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64\": rpc error: code = NotFound desc = could not find container \"fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64\": container with ID starting with fd4c3329f53b098dbd357084cda55a8ccd833cca04fc0bb2b5177d66a0fbee64 not found: ID does not exist"
Sep 11 14:32:32.002524 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.002465    2558 scope.go:117] "RemoveContainer" containerID="8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523"
Sep 11 14:32:32.010075 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.010060    2558 scope.go:117] "RemoveContainer" containerID="8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523"
Sep 11 14:32:32.010333 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:32:32.010315    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523\": container with ID starting with 8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523 not found: ID does not exist" containerID="8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523"
Sep 11 14:32:32.010390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.010337    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523"} err="failed to get container status \"8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523\": rpc error: code = NotFound desc = could not find container \"8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523\": container with ID starting with 8d0534e9bdff428b3daac2b00791c3afdece73b39d203c97037ed1755fa8c523 not found: ID does not exist"
Sep 11 14:32:32.021621 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.021597    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"]
Sep 11 14:32:32.028117 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.028098    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1e2e7d-79c995695f-lz7k7"]
Sep 11 14:32:32.044489 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.044464    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"]
Sep 11 14:32:32.049752 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.049731    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1e2e7d-6dd8475666-flg8c"]
Sep 11 14:32:32.245082 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.244993    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:32.397793 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.397762    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:32.502471 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.502401    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca15502-7c36-491c-afab-96654299f89d" path="/var/lib/kubelet/pods/8ca15502-7c36-491c-afab-96654299f89d/volumes"
Sep 11 14:32:32.502898 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.502878    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a186b216-4532-44a4-8f51-56ca6ccf4439" path="/var/lib/kubelet/pods/a186b216-4532-44a4-8f51-56ca6ccf4439/volumes"
Sep 11 14:32:32.503266 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:32.503247    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d591fbf9-3720-4475-bb45-2387c8abf465" path="/var/lib/kubelet/pods/d591fbf9-3720-4475-bb45-2387c8abf465/volumes"
Sep 11 14:32:33.427828 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:33.427797    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:34.667382 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:34.667352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:32:39.171588 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.171551    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:39.293388 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.293362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:39.546285 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.546256    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:39.891960 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.891894    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"]
Sep 11 14:32:39.892234 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892222    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca15502-7c36-491c-afab-96654299f89d" containerName="payload-processing"
Sep 11 14:32:39.892281 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892236    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca15502-7c36-491c-afab-96654299f89d" containerName="payload-processing"
Sep 11 14:32:39.892281 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892246    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a186b216-4532-44a4-8f51-56ca6ccf4439" containerName="payload-pre-processing"
Sep 11 14:32:39.892281 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892252    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a186b216-4532-44a4-8f51-56ca6ccf4439" containerName="payload-pre-processing"
Sep 11 14:32:39.892281 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892259    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d591fbf9-3720-4475-bb45-2387c8abf465" containerName="revoke-keys"
Sep 11 14:32:39.892281 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892264    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d591fbf9-3720-4475-bb45-2387c8abf465" containerName="revoke-keys"
Sep 11 14:32:39.892430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892322    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a186b216-4532-44a4-8f51-56ca6ccf4439" containerName="payload-pre-processing"
Sep 11 14:32:39.892430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892332    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d591fbf9-3720-4475-bb45-2387c8abf465" containerName="revoke-keys"
Sep 11 14:32:39.892430 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.892340    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca15502-7c36-491c-afab-96654299f89d" containerName="payload-processing"
Sep 11 14:32:39.896778 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.896759    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:39.909085 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.909064    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:32:39.909188 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.909111    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\""
Sep 11 14:32:39.936613 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.936593    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"]
Sep 11 14:32:39.951540 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:39.951514    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308c66a6-602f-4a23-81c6-418e81bd8547-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd\" (UID: \"308c66a6-602f-4a23-81c6-418e81bd8547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:40.052136 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:40.052107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308c66a6-602f-4a23-81c6-418e81bd8547-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd\" (UID: \"308c66a6-602f-4a23-81c6-418e81bd8547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:40.052678 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:40.052661    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308c66a6-602f-4a23-81c6-418e81bd8547-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd\" (UID: \"308c66a6-602f-4a23-81c6-418e81bd8547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:40.206864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:40.206840    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:40.486898 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:40.486874    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"]
Sep 11 14:32:40.489487 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:32:40.489457    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308c66a6_602f_4a23_81c6_418e81bd8547.slice/crio-e4f27714deb4de36b5a05dd69fd58b08a3abec876a86a0839bfb04e837efb595 WatchSource:0}: Error finding container e4f27714deb4de36b5a05dd69fd58b08a3abec876a86a0839bfb04e837efb595: Status 404 returned error can't find the container with id e4f27714deb4de36b5a05dd69fd58b08a3abec876a86a0839bfb04e837efb595
Sep 11 14:32:41.040742 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:41.040707    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd" event={"ID":"308c66a6-602f-4a23-81c6-418e81bd8547","Type":"ContainerStarted","Data":"9aa804b87ecbc3f3088405ff7fe31c5ad4f7c099b1528e930a7fdb4c4fb84133"}
Sep 11 14:32:41.040903 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:41.040796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd" event={"ID":"308c66a6-602f-4a23-81c6-418e81bd8547","Type":"ContainerStarted","Data":"e4f27714deb4de36b5a05dd69fd58b08a3abec876a86a0839bfb04e837efb595"}
Sep 11 14:32:41.076065 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:41.076004    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd" podStartSLOduration=2.075991545 podStartE2EDuration="2.075991545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:41.073504035 +0000 UTC m=+2095.037119007" watchObservedRunningTime="2026-09-11 14:32:41.075991545 +0000 UTC m=+2095.039606517"
Sep 11 14:32:41.124168 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:41.124133    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:42.034338 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:42.034302    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:42.047407 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:42.047380    2558 generic.go:356] "Generic (PLEG): container finished" podID="308c66a6-602f-4a23-81c6-418e81bd8547" containerID="9aa804b87ecbc3f3088405ff7fe31c5ad4f7c099b1528e930a7fdb4c4fb84133" exitCode=0
Sep 11 14:32:42.047519 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:42.047457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd" event={"ID":"308c66a6-602f-4a23-81c6-418e81bd8547","Type":"ContainerDied","Data":"9aa804b87ecbc3f3088405ff7fe31c5ad4f7c099b1528e930a7fdb4c4fb84133"}
Sep 11 14:32:43.173622 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:43.173598    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:43.278724 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:43.278694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308c66a6-602f-4a23-81c6-418e81bd8547-maas-api-service-ca\") pod \"308c66a6-602f-4a23-81c6-418e81bd8547\" (UID: \"308c66a6-602f-4a23-81c6-418e81bd8547\") "
Sep 11 14:32:43.279004 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:43.278984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308c66a6-602f-4a23-81c6-418e81bd8547-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "308c66a6-602f-4a23-81c6-418e81bd8547" (UID: "308c66a6-602f-4a23-81c6-418e81bd8547"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:43.379950 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:43.379897    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/308c66a6-602f-4a23-81c6-418e81bd8547-maas-api-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:44.057150 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:44.057125    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"
Sep 11 14:32:44.057150 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:44.057141    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd" event={"ID":"308c66a6-602f-4a23-81c6-418e81bd8547","Type":"ContainerDied","Data":"e4f27714deb4de36b5a05dd69fd58b08a3abec876a86a0839bfb04e837efb595"}
Sep 11 14:32:44.057363 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:44.057182    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f27714deb4de36b5a05dd69fd58b08a3abec876a86a0839bfb04e837efb595"
Sep 11 14:32:46.588113 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:46.588082    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:32:46.592900 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:46.592880    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:32:50.655414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.655379    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:50.717884 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.717849    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"]
Sep 11 14:32:50.718185 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.718060    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff" podUID="79b9f040-38a5-4367-9bb1-678e7324fc5c" containerName="payload-processing" containerID="cri-o://05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4" gracePeriod=30
Sep 11 14:32:50.770609 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.770570    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:50.873071 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.873021    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:50.960053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.960012    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:32:50.971018 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.970989    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"]
Sep 11 14:32:50.977652 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.977631    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-031a56-2hvtd"]
Sep 11 14:32:50.985312 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:50.985287    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:51.043821 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.043798    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbsv6\" (UniqueName: \"kubernetes.io/projected/79b9f040-38a5-4367-9bb1-678e7324fc5c-kube-api-access-fbsv6\") pod \"79b9f040-38a5-4367-9bb1-678e7324fc5c\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") "
Sep 11 14:32:51.043918 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.043834    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9f040-38a5-4367-9bb1-678e7324fc5c-plugins-config-volume\") pod \"79b9f040-38a5-4367-9bb1-678e7324fc5c\" (UID: \"79b9f040-38a5-4367-9bb1-678e7324fc5c\") "
Sep 11 14:32:51.044235 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.044204    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b9f040-38a5-4367-9bb1-678e7324fc5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79b9f040-38a5-4367-9bb1-678e7324fc5c" (UID: "79b9f040-38a5-4367-9bb1-678e7324fc5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:51.045837 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.045807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b9f040-38a5-4367-9bb1-678e7324fc5c-kube-api-access-fbsv6" (OuterVolumeSpecName: "kube-api-access-fbsv6") pod "79b9f040-38a5-4367-9bb1-678e7324fc5c" (UID: "79b9f040-38a5-4367-9bb1-678e7324fc5c"). InnerVolumeSpecName "kube-api-access-fbsv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:51.082141 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.082114    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:51.088615 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.088595    2558 generic.go:356] "Generic (PLEG): container finished" podID="79b9f040-38a5-4367-9bb1-678e7324fc5c" containerID="05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4" exitCode=0
Sep 11 14:32:51.088719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.088634    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"
Sep 11 14:32:51.088719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.088674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff" event={"ID":"79b9f040-38a5-4367-9bb1-678e7324fc5c","Type":"ContainerDied","Data":"05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4"}
Sep 11 14:32:51.088719 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.088718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff" event={"ID":"79b9f040-38a5-4367-9bb1-678e7324fc5c","Type":"ContainerDied","Data":"e5edcbdd28f545be390ff47dcd90190ee853cfb10efebc5ec7d4a1d9bcaadbc2"}
Sep 11 14:32:51.088852 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.088739    2558 scope.go:117] "RemoveContainer" containerID="05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4"
Sep 11 14:32:51.097291 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.097274    2558 scope.go:117] "RemoveContainer" containerID="05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4"
Sep 11 14:32:51.097553 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:32:51.097535    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4\": container with ID starting with 05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4 not found: ID does not exist" containerID="05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4"
Sep 11 14:32:51.097608 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.097566    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4"} err="failed to get container status \"05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4\": rpc error: code = NotFound desc = could not find container \"05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4\": container with ID starting with 05b185c6681755f9bf93b4cac08caa275cde8f1eda0b1e350858fd398c0987b4 not found: ID does not exist"
Sep 11 14:32:51.111045 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.111013    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"]
Sep 11 14:32:51.120126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.120100    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-031a56-7dccc75645-7jrff"]
Sep 11 14:32:51.145401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.145383    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbsv6\" (UniqueName: \"kubernetes.io/projected/79b9f040-38a5-4367-9bb1-678e7324fc5c-kube-api-access-fbsv6\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:51.145479 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.145401    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9f040-38a5-4367-9bb1-678e7324fc5c-plugins-config-volume\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:32:51.384167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.384084    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:51.495099 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:51.495072    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:52.503177 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:52.503145    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="308c66a6-602f-4a23-81c6-418e81bd8547" path="/var/lib/kubelet/pods/308c66a6-602f-4a23-81c6-418e81bd8547/volumes"
Sep 11 14:32:52.503529 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:52.503471    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b9f040-38a5-4367-9bb1-678e7324fc5c" path="/var/lib/kubelet/pods/79b9f040-38a5-4367-9bb1-678e7324fc5c/volumes"
Sep 11 14:32:53.300596 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:53.300560    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:32:57.817307 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:57.817276    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:57.899080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:57.899055    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:32:58.120149 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:32:58.120053    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:33:02.162414 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:33:02.162380    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:33:21.666114 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:33:21.666077    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:33:46.163559 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:33:46.163462    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:34:22.789186 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:22.789145    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:34:23.988461 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:23.988417    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:34:24.135605 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:24.135568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:34:24.194011 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:24.193982    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:34:25.495790 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:25.495744    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:34:47.681295 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:47.681259    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:34:48.964959 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:48.964926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:34:55.675262 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:55.675219    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:34:58.813654 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:34:58.813620    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:35:03.734989 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:35:03.734959    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:35:04.170080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:35:04.169970    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:35:24.074345 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:35:24.074309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:35:29.864010 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:35:29.863975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:36:46.836573 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:36:46.836494    2558 scope.go:117] "RemoveContainer" containerID="c81e5f66fd6d97c2907a28a6df1a0ebd2167216f217e20867c4a52bcecb6b1f3"
Sep 11 14:36:56.574573 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:36:56.574541    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:05.567616 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:05.567576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:06.952153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952123    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"]
Sep 11 14:37:06.952651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952479    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b9f040-38a5-4367-9bb1-678e7324fc5c" containerName="payload-processing"
Sep 11 14:37:06.952651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952501    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b9f040-38a5-4367-9bb1-678e7324fc5c" containerName="payload-processing"
Sep 11 14:37:06.952651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952510    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308c66a6-602f-4a23-81c6-418e81bd8547" containerName="revoke-keys"
Sep 11 14:37:06.952651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952515    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="308c66a6-602f-4a23-81c6-418e81bd8547" containerName="revoke-keys"
Sep 11 14:37:06.952651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952590    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="308c66a6-602f-4a23-81c6-418e81bd8547" containerName="revoke-keys"
Sep 11 14:37:06.952651 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.952604    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b9f040-38a5-4367-9bb1-678e7324fc5c" containerName="payload-processing"
Sep 11 14:37:06.955583 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.955567    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:06.958541 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.958520    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9a20169b-kserve-self-signed-certs\""
Sep 11 14:37:06.970356 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:06.970331    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"]
Sep 11 14:37:07.036785 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.036754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8m25\" (UniqueName: \"kubernetes.io/projected/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-kube-api-access-l8m25\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.036920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.036792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tmp-dir\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.036920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.036811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tls-certs\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.036920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.036892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-model-cache\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.037080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.036954    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-home\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.037080 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.036974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-dshm\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.110243 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.110215    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:37:07.137383 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tls-certs\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.137477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-model-cache\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.137477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-home\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.137477 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-dshm\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.137597 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8m25\" (UniqueName: \"kubernetes.io/projected/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-kube-api-access-l8m25\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.137597 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tmp-dir\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.137926 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-model-cache\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.138020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137945    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-home\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.138020 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.137953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tmp-dir\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.139717 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.139689    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-dshm\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.139939 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.139921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tls-certs\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.156782 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.156753    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8m25\" (UniqueName: \"kubernetes.io/projected/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-kube-api-access-l8m25\") pod \"e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") " pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.265092 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.265003    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:07.539465 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:07.539441    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"]
Sep 11 14:37:07.541892 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:37:07.541862    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2bc98b1_9ed5_4444_a507_f34bc2da16e0.slice/crio-4ee6ff966bbbf34ef6b405f9bde47ed62946c129e7d47469a4528d82e1b8054d WatchSource:0}: Error finding container 4ee6ff966bbbf34ef6b405f9bde47ed62946c129e7d47469a4528d82e1b8054d: Status 404 returned error can't find the container with id 4ee6ff966bbbf34ef6b405f9bde47ed62946c129e7d47469a4528d82e1b8054d
Sep 11 14:37:08.238224 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:08.238190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj" event={"ID":"d2bc98b1-9ed5-4444-a507-f34bc2da16e0","Type":"ContainerStarted","Data":"bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05"}
Sep 11 14:37:08.238224 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:08.238229    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj" event={"ID":"d2bc98b1-9ed5-4444-a507-f34bc2da16e0","Type":"ContainerStarted","Data":"4ee6ff966bbbf34ef6b405f9bde47ed62946c129e7d47469a4528d82e1b8054d"}
Sep 11 14:37:08.257487 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:08.257443    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj" podStartSLOduration=2.257427873 podStartE2EDuration="2.257427873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:08.256565947 +0000 UTC m=+2362.220180920" watchObservedRunningTime="2026-09-11 14:37:08.257427873 +0000 UTC m=+2362.221042880"
Sep 11 14:37:14.563839 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:14.563796    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:17.265453 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:17.265417    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:17.265855 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:17.265464    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:17.277994 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:17.277967    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:17.288851 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:17.288830    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:22.061351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.061320    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"]
Sep 11 14:37:22.064809 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.064783    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.067571 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.067552    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9a20169b-kserve-self-signed-certs\""
Sep 11 14:37:22.079626 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.079604    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"]
Sep 11 14:37:22.166066 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.166019    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-tmp-dir\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.166199 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.166082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-home\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.166199 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.166149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-model-cache\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.166323 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.166215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da39b63-b76d-44c0-a659-ce741d703f41-tls-certs\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.166323 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.166254    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tws49\" (UniqueName: \"kubernetes.io/projected/3da39b63-b76d-44c0-a659-ce741d703f41-kube-api-access-tws49\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.166323 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.166282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-dshm\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.205496 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.205475    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:37:22.267587 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-tmp-dir\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.267696 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-home\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.267696 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-model-cache\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.267696 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da39b63-b76d-44c0-a659-ce741d703f41-tls-certs\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.267696 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tws49\" (UniqueName: \"kubernetes.io/projected/3da39b63-b76d-44c0-a659-ce741d703f41-kube-api-access-tws49\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.267696 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-dshm\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.267991 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.267966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-tmp-dir\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.268075 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.268053    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-model-cache\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.268126 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.268094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-home\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.269887 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.269862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-dshm\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.270120 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.270103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da39b63-b76d-44c0-a659-ce741d703f41-tls-certs\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.279316 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.279295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tws49\" (UniqueName: \"kubernetes.io/projected/3da39b63-b76d-44c0-a659-ce741d703f41-kube-api-access-tws49\") pod \"e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") " pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.373569 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.373504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:22.646929 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:22.646905    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"]
Sep 11 14:37:22.648403 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:37:22.648375    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3da39b63_b76d_44c0_a659_ce741d703f41.slice/crio-a966d8375be254c9b29bf280741d9e626abb2069b7378d480788ba8b34c4f61c WatchSource:0}: Error finding container a966d8375be254c9b29bf280741d9e626abb2069b7378d480788ba8b34c4f61c: Status 404 returned error can't find the container with id a966d8375be254c9b29bf280741d9e626abb2069b7378d480788ba8b34c4f61c
Sep 11 14:37:23.303042 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:23.302993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj" event={"ID":"3da39b63-b76d-44c0-a659-ce741d703f41","Type":"ContainerStarted","Data":"99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0"}
Sep 11 14:37:23.303495 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:23.303058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj" event={"ID":"3da39b63-b76d-44c0-a659-ce741d703f41","Type":"ContainerStarted","Data":"a966d8375be254c9b29bf280741d9e626abb2069b7378d480788ba8b34c4f61c"}
Sep 11 14:37:23.328408 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:23.328358    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj" podStartSLOduration=1.328345207 podStartE2EDuration="1.328345207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:23.326652279 +0000 UTC m=+2377.290267251" watchObservedRunningTime="2026-09-11 14:37:23.328345207 +0000 UTC m=+2377.291960178"
Sep 11 14:37:24.981864 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:24.981820    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:32.374015 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:32.373980    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:32.374015 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:32.374038    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:32.386304 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:32.386281    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:33.355957 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:33.355927    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:40.867737 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:40.867704    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"]
Sep 11 14:37:40.894312 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:40.894284    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:37:42.076335 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:42.076302    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"]
Sep 11 14:37:42.368872 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:42.368783    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:37:43.765270 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:43.765233    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:46.617455 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:46.617425    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:37:46.630167 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:46.630142    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dsrq8_a6922043-21b1-4117-90b6-8a6b54f09902/ovn-acl-logging/0.log"
Sep 11 14:37:51.369819 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:51.369783    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:55.885883 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:55.885824    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj" podUID="d2bc98b1-9ed5-4444-a507-f34bc2da16e0" containerName="main" containerID="cri-o://bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05" gracePeriod=45
Sep 11 14:37:56.122646 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.122620    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"]
Sep 11 14:37:56.126003 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.125986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.129426 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.129399    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:37:56.129554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.129407    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 14:37:56.129554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.129441    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:37:56.129554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.129442    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktm9c\""
Sep 11 14:37:56.129554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.129455    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:37:56.134816 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.134795    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:56.142497 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.142474    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"]
Sep 11 14:37:56.263155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263123    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tmp-dir\") pod \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") "
Sep 11 14:37:56.263325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263162    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tls-certs\") pod \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") "
Sep 11 14:37:56.263325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263217    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-home\") pod \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") "
Sep 11 14:37:56.263325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263262    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8m25\" (UniqueName: \"kubernetes.io/projected/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-kube-api-access-l8m25\") pod \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") "
Sep 11 14:37:56.263325 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263281    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-model-cache\") pod \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") "
Sep 11 14:37:56.263498 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263346    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-dshm\") pod \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\" (UID: \"d2bc98b1-9ed5-4444-a507-f34bc2da16e0\") "
Sep 11 14:37:56.263498 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263421    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2bc98b1-9ed5-4444-a507-f34bc2da16e0" (UID: "d2bc98b1-9ed5-4444-a507-f34bc2da16e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:56.263610 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-maas-api-tls\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.263610 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263549    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-home" (OuterVolumeSpecName: "home") pod "d2bc98b1-9ed5-4444-a507-f34bc2da16e0" (UID: "d2bc98b1-9ed5-4444-a507-f34bc2da16e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:56.263610 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263563    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2bc98b1-9ed5-4444-a507-f34bc2da16e0" (UID: "d2bc98b1-9ed5-4444-a507-f34bc2da16e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:56.263767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62410c2-a634-4546-957e-30d3b5c30689-openshift-service-ca\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.263767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263662    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-metrics-certs\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.263767 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dbvm\" (UniqueName: \"kubernetes.io/projected/c62410c2-a634-4546-957e-30d3b5c30689-kube-api-access-9dbvm\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.263868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263799    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:56.263868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263819    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:56.263868 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.263830    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:56.265381 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.265348    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-dshm" (OuterVolumeSpecName: "dshm") pod "d2bc98b1-9ed5-4444-a507-f34bc2da16e0" (UID: "d2bc98b1-9ed5-4444-a507-f34bc2da16e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:56.265507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.265406    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2bc98b1-9ed5-4444-a507-f34bc2da16e0" (UID: "d2bc98b1-9ed5-4444-a507-f34bc2da16e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:37:56.265507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.265413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-kube-api-access-l8m25" (OuterVolumeSpecName: "kube-api-access-l8m25") pod "d2bc98b1-9ed5-4444-a507-f34bc2da16e0" (UID: "d2bc98b1-9ed5-4444-a507-f34bc2da16e0"). InnerVolumeSpecName "kube-api-access-l8m25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:37:56.364280 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-maas-api-tls\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.364450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62410c2-a634-4546-957e-30d3b5c30689-openshift-service-ca\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.364450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-metrics-certs\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.364450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dbvm\" (UniqueName: \"kubernetes.io/projected/c62410c2-a634-4546-957e-30d3b5c30689-kube-api-access-9dbvm\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.364450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364399    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8m25\" (UniqueName: \"kubernetes.io/projected/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-kube-api-access-l8m25\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:56.364450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364408    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:56.364450 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.364416    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2bc98b1-9ed5-4444-a507-f34bc2da16e0-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:56.365097 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.365071    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62410c2-a634-4546-957e-30d3b5c30689-openshift-service-ca\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.366867 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.366845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-maas-api-tls\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.366952 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.366911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-metrics-certs\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.372825 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.372804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dbvm\" (UniqueName: \"kubernetes.io/projected/c62410c2-a634-4546-957e-30d3b5c30689-kube-api-access-9dbvm\") pod \"maas-api-d8d795dc5-nljg4\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") " pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.439824 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.439795    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:56.451334 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.451310    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2bc98b1-9ed5-4444-a507-f34bc2da16e0" containerID="bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05" exitCode=0
Sep 11 14:37:56.451412 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.451348    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"
Sep 11 14:37:56.451412 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.451385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj" event={"ID":"d2bc98b1-9ed5-4444-a507-f34bc2da16e0","Type":"ContainerDied","Data":"bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05"}
Sep 11 14:37:56.451515 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.451429    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj" event={"ID":"d2bc98b1-9ed5-4444-a507-f34bc2da16e0","Type":"ContainerDied","Data":"4ee6ff966bbbf34ef6b405f9bde47ed62946c129e7d47469a4528d82e1b8054d"}
Sep 11 14:37:56.451515 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.451448    2558 scope.go:117] "RemoveContainer" containerID="bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05"
Sep 11 14:37:56.461275 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.461255    2558 scope.go:117] "RemoveContainer" containerID="bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05"
Sep 11 14:37:56.461583 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:37:56.461563    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05\": container with ID starting with bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05 not found: ID does not exist" containerID="bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05"
Sep 11 14:37:56.461630 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.461591    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05"} err="failed to get container status \"bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05\": rpc error: code = NotFound desc = could not find container \"bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05\": container with ID starting with bcf59ce2c23bfbb254bdd856351b8f1c2606ee3a26767da5426fc0d0bdd3fa05 not found: ID does not exist"
Sep 11 14:37:56.476202 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.476174    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"]
Sep 11 14:37:56.480172 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.480145    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9a20169b-kserve-8494cd7d4c-445dj"]
Sep 11 14:37:56.504627 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.504602    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2bc98b1-9ed5-4444-a507-f34bc2da16e0" path="/var/lib/kubelet/pods/d2bc98b1-9ed5-4444-a507-f34bc2da16e0/volumes"
Sep 11 14:37:56.674003 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.673970    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:37:56.716981 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.716958    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"]
Sep 11 14:37:56.719541 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:37:56.719518    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62410c2_a634_4546_957e_30d3b5c30689.slice/crio-c7897df8eb56998b63501ed608407c8fcd62bbb7ddf29e5408ebba3f00f6e1f1 WatchSource:0}: Error finding container c7897df8eb56998b63501ed608407c8fcd62bbb7ddf29e5408ebba3f00f6e1f1: Status 404 returned error can't find the container with id c7897df8eb56998b63501ed608407c8fcd62bbb7ddf29e5408ebba3f00f6e1f1
Sep 11 14:37:56.720834 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:56.720812    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:37:57.094511 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.094414    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj" podUID="3da39b63-b76d-44c0-a659-ce741d703f41" containerName="main" containerID="cri-o://99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0" gracePeriod=45
Sep 11 14:37:57.341401 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.341381    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:57.464168 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.464130    2558 generic.go:356] "Generic (PLEG): container finished" podID="3da39b63-b76d-44c0-a659-ce741d703f41" containerID="99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0" exitCode=0
Sep 11 14:37:57.464351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.464183    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"
Sep 11 14:37:57.464351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.464210    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj" event={"ID":"3da39b63-b76d-44c0-a659-ce741d703f41","Type":"ContainerDied","Data":"99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0"}
Sep 11 14:37:57.464351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.464252    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj" event={"ID":"3da39b63-b76d-44c0-a659-ce741d703f41","Type":"ContainerDied","Data":"a966d8375be254c9b29bf280741d9e626abb2069b7378d480788ba8b34c4f61c"}
Sep 11 14:37:57.464351 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.464269    2558 scope.go:117] "RemoveContainer" containerID="99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0"
Sep 11 14:37:57.465874 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.465841    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4" event={"ID":"c62410c2-a634-4546-957e-30d3b5c30689","Type":"ContainerStarted","Data":"d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d"}
Sep 11 14:37:57.465978 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.465906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4" event={"ID":"c62410c2-a634-4546-957e-30d3b5c30689","Type":"ContainerStarted","Data":"c7897df8eb56998b63501ed608407c8fcd62bbb7ddf29e5408ebba3f00f6e1f1"}
Sep 11 14:37:57.465978 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.465926    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:37:57.472458 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472439    2558 scope.go:117] "RemoveContainer" containerID="99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0"
Sep 11 14:37:57.472709 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:37:57.472688    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0\": container with ID starting with 99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0 not found: ID does not exist" containerID="99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0"
Sep 11 14:37:57.472788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472718    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-tmp-dir\") pod \"3da39b63-b76d-44c0-a659-ce741d703f41\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") "
Sep 11 14:37:57.472788 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472770    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-dshm\") pod \"3da39b63-b76d-44c0-a659-ce741d703f41\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") "
Sep 11 14:37:57.472892 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472828    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-model-cache\") pod \"3da39b63-b76d-44c0-a659-ce741d703f41\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") "
Sep 11 14:37:57.472892 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472716    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0"} err="failed to get container status \"99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0\": rpc error: code = NotFound desc = could not find container \"99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0\": container with ID starting with 99142b54ce3e8db6dbf672f7dfa539242fe98b882f176bcc4cfa6a5e227ea4f0 not found: ID does not exist"
Sep 11 14:37:57.472892 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472872    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-home\") pod \"3da39b63-b76d-44c0-a659-ce741d703f41\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") "
Sep 11 14:37:57.473069 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472911    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3da39b63-b76d-44c0-a659-ce741d703f41" (UID: "3da39b63-b76d-44c0-a659-ce741d703f41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:57.473069 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.472921    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tws49\" (UniqueName: \"kubernetes.io/projected/3da39b63-b76d-44c0-a659-ce741d703f41-kube-api-access-tws49\") pod \"3da39b63-b76d-44c0-a659-ce741d703f41\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") "
Sep 11 14:37:57.473069 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.473046    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da39b63-b76d-44c0-a659-ce741d703f41-tls-certs\") pod \"3da39b63-b76d-44c0-a659-ce741d703f41\" (UID: \"3da39b63-b76d-44c0-a659-ce741d703f41\") "
Sep 11 14:37:57.473229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.473128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-model-cache" (OuterVolumeSpecName: "model-cache") pod "3da39b63-b76d-44c0-a659-ce741d703f41" (UID: "3da39b63-b76d-44c0-a659-ce741d703f41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:57.473229 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.473141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-home" (OuterVolumeSpecName: "home") pod "3da39b63-b76d-44c0-a659-ce741d703f41" (UID: "3da39b63-b76d-44c0-a659-ce741d703f41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:57.473394 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.473375    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-tmp-dir\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:57.473446 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.473397    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-model-cache\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:57.473446 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.473409    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-home\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:57.474660 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.474641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-dshm" (OuterVolumeSpecName: "dshm") pod "3da39b63-b76d-44c0-a659-ce741d703f41" (UID: "3da39b63-b76d-44c0-a659-ce741d703f41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:37:57.474908 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.474888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da39b63-b76d-44c0-a659-ce741d703f41-kube-api-access-tws49" (OuterVolumeSpecName: "kube-api-access-tws49") pod "3da39b63-b76d-44c0-a659-ce741d703f41" (UID: "3da39b63-b76d-44c0-a659-ce741d703f41"). InnerVolumeSpecName "kube-api-access-tws49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:37:57.474955 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.474918    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da39b63-b76d-44c0-a659-ce741d703f41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3da39b63-b76d-44c0-a659-ce741d703f41" (UID: "3da39b63-b76d-44c0-a659-ce741d703f41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:37:57.486447 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.486409    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4" podStartSLOduration=0.892607845 podStartE2EDuration="1.486398324s" totalImagesPullingTime="593.790479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:57.485553621 +0000 UTC m=+2411.449168617" watchObservedRunningTime="2026-09-11 14:37:57.486398324 +0000 UTC m=+2411.450013297"
Sep 11 14:37:57.574371 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.574334    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3da39b63-b76d-44c0-a659-ce741d703f41-dshm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:57.574371 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.574365    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tws49\" (UniqueName: \"kubernetes.io/projected/3da39b63-b76d-44c0-a659-ce741d703f41-kube-api-access-tws49\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:57.574371 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.574379    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3da39b63-b76d-44c0-a659-ce741d703f41-tls-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:37:57.792514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.792481    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"]
Sep 11 14:37:57.799920 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:57.799898    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9a20169b-kserve-59f477d87f-hm9lj"]
Sep 11 14:37:58.502721 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:37:58.502690    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3da39b63-b76d-44c0-a659-ce741d703f41" path="/var/lib/kubelet/pods/3da39b63-b76d-44c0-a659-ce741d703f41/volumes"
Sep 11 14:38:03.476904 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.476872    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:38:03.566525 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.566496    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:38:03.840046 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.839945    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"]
Sep 11 14:38:03.840329 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.840315    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2bc98b1-9ed5-4444-a507-f34bc2da16e0" containerName="main"
Sep 11 14:38:03.840372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.840331    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2bc98b1-9ed5-4444-a507-f34bc2da16e0" containerName="main"
Sep 11 14:38:03.840372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.840341    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da39b63-b76d-44c0-a659-ce741d703f41" containerName="main"
Sep 11 14:38:03.840372 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.840346    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da39b63-b76d-44c0-a659-ce741d703f41" containerName="main"
Sep 11 14:38:03.840465 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.840405    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2bc98b1-9ed5-4444-a507-f34bc2da16e0" containerName="main"
Sep 11 14:38:03.840465 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.840415    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da39b63-b76d-44c0-a659-ce741d703f41" containerName="main"
Sep 11 14:38:03.844387 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.844371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:03.855741 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.855716    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"]
Sep 11 14:38:03.930327 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.930298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d605701a-788b-44d0-935d-81bd0b75df29-metrics-certs\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:03.930327 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.930328    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkrb\" (UniqueName: \"kubernetes.io/projected/d605701a-788b-44d0-935d-81bd0b75df29-kube-api-access-7nkrb\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:03.930512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.930375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d605701a-788b-44d0-935d-81bd0b75df29-maas-api-tls\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:03.930512 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:03.930485    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d605701a-788b-44d0-935d-81bd0b75df29-openshift-service-ca\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.031197 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.031170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d605701a-788b-44d0-935d-81bd0b75df29-openshift-service-ca\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.031353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.031214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d605701a-788b-44d0-935d-81bd0b75df29-metrics-certs\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.031353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.031232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkrb\" (UniqueName: \"kubernetes.io/projected/d605701a-788b-44d0-935d-81bd0b75df29-kube-api-access-7nkrb\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.031353 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.031284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d605701a-788b-44d0-935d-81bd0b75df29-maas-api-tls\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.031925 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.031902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d605701a-788b-44d0-935d-81bd0b75df29-openshift-service-ca\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.033671 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.033643    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d605701a-788b-44d0-935d-81bd0b75df29-maas-api-tls\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.033765 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.033695    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d605701a-788b-44d0-935d-81bd0b75df29-metrics-certs\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.040354 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.040336    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkrb\" (UniqueName: \"kubernetes.io/projected/d605701a-788b-44d0-935d-81bd0b75df29-kube-api-access-7nkrb\") pod \"maas-api-7959f48cf9-drqs6\" (UID: \"d605701a-788b-44d0-935d-81bd0b75df29\") " pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.154543 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.154476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:04.426493 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.426413    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"]
Sep 11 14:38:04.430280 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:38:04.430249    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd605701a_788b_44d0_935d_81bd0b75df29.slice/crio-4944442646c6eba42ee28ce02010ab2aff2f3d8cc83765ea4083f3c0ecd41856 WatchSource:0}: Error finding container 4944442646c6eba42ee28ce02010ab2aff2f3d8cc83765ea4083f3c0ecd41856: Status 404 returned error can't find the container with id 4944442646c6eba42ee28ce02010ab2aff2f3d8cc83765ea4083f3c0ecd41856
Sep 11 14:38:04.502429 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:04.502399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6" event={"ID":"d605701a-788b-44d0-935d-81bd0b75df29","Type":"ContainerStarted","Data":"4944442646c6eba42ee28ce02010ab2aff2f3d8cc83765ea4083f3c0ecd41856"}
Sep 11 14:38:05.503756 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:05.503715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6" event={"ID":"d605701a-788b-44d0-935d-81bd0b75df29","Type":"ContainerStarted","Data":"25374f7487ac75bb28ea610b6cc3764587ff2903600fc7b65e414182f12387f7"}
Sep 11 14:38:05.504124 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:05.503844    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:05.526746 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:05.526694    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6" podStartSLOduration=1.586757572 podStartE2EDuration="2.526678066s" totalImagesPullingTime="939.920494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:05.525992162 +0000 UTC m=+2419.489607134" watchObservedRunningTime="2026-09-11 14:38:05.526678066 +0000 UTC m=+2419.490293039"
Sep 11 14:38:10.481436 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:10.481222    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:38:11.514246 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:11.514219    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7959f48cf9-drqs6"
Sep 11 14:38:11.571340 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:11.571301    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"]
Sep 11 14:38:11.571557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:11.571534    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4" podUID="c62410c2-a634-4546-957e-30d3b5c30689" containerName="maas-api" containerID="cri-o://d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d" gracePeriod=30
Sep 11 14:38:12.809169 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.809144    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:38:12.909915 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.909883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dbvm\" (UniqueName: \"kubernetes.io/projected/c62410c2-a634-4546-957e-30d3b5c30689-kube-api-access-9dbvm\") pod \"c62410c2-a634-4546-957e-30d3b5c30689\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") "
Sep 11 14:38:12.910105 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.909936    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62410c2-a634-4546-957e-30d3b5c30689-openshift-service-ca\") pod \"c62410c2-a634-4546-957e-30d3b5c30689\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") "
Sep 11 14:38:12.910105 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.909969    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-metrics-certs\") pod \"c62410c2-a634-4546-957e-30d3b5c30689\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") "
Sep 11 14:38:12.910105 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.910005    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-maas-api-tls\") pod \"c62410c2-a634-4546-957e-30d3b5c30689\" (UID: \"c62410c2-a634-4546-957e-30d3b5c30689\") "
Sep 11 14:38:12.910323 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.910295    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62410c2-a634-4546-957e-30d3b5c30689-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c62410c2-a634-4546-957e-30d3b5c30689" (UID: "c62410c2-a634-4546-957e-30d3b5c30689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:38:12.910399 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.910383    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62410c2-a634-4546-957e-30d3b5c30689-openshift-service-ca\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:38:12.912211 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.912181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c62410c2-a634-4546-957e-30d3b5c30689" (UID: "c62410c2-a634-4546-957e-30d3b5c30689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:38:12.912466 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.912445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c62410c2-a634-4546-957e-30d3b5c30689" (UID: "c62410c2-a634-4546-957e-30d3b5c30689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:38:12.912534 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:12.912465    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62410c2-a634-4546-957e-30d3b5c30689-kube-api-access-9dbvm" (OuterVolumeSpecName: "kube-api-access-9dbvm") pod "c62410c2-a634-4546-957e-30d3b5c30689" (UID: "c62410c2-a634-4546-957e-30d3b5c30689"). InnerVolumeSpecName "kube-api-access-9dbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:38:13.010956 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.010917    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dbvm\" (UniqueName: \"kubernetes.io/projected/c62410c2-a634-4546-957e-30d3b5c30689-kube-api-access-9dbvm\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:38:13.010956 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.010953    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-metrics-certs\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:38:13.010956 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.010962    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c62410c2-a634-4546-957e-30d3b5c30689-maas-api-tls\") on node \"ip-10-0-140-144.ec2.internal\" DevicePath \"\""
Sep 11 14:38:13.541845 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.541808    2558 generic.go:356] "Generic (PLEG): container finished" podID="c62410c2-a634-4546-957e-30d3b5c30689" containerID="d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d" exitCode=0
Sep 11 14:38:13.542053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.541856    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"
Sep 11 14:38:13.542053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.541896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4" event={"ID":"c62410c2-a634-4546-957e-30d3b5c30689","Type":"ContainerDied","Data":"d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d"}
Sep 11 14:38:13.542053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.541949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4" event={"ID":"c62410c2-a634-4546-957e-30d3b5c30689","Type":"ContainerDied","Data":"c7897df8eb56998b63501ed608407c8fcd62bbb7ddf29e5408ebba3f00f6e1f1"}
Sep 11 14:38:13.542053 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.541979    2558 scope.go:117] "RemoveContainer" containerID="d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d"
Sep 11 14:38:13.550758 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.550744    2558 scope.go:117] "RemoveContainer" containerID="d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d"
Sep 11 14:38:13.551019 ip-10-0-140-144 kubenswrapper[2558]: E0911 14:38:13.551001    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d\": container with ID starting with d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d not found: ID does not exist" containerID="d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d"
Sep 11 14:38:13.551105 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.551040    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d"} err="failed to get container status \"d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d\": rpc error: code = NotFound desc = could not find container \"d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d\": container with ID starting with d91f969503febca05e48a4f9046e35df170b029384acbc664836ca09c0064b6d not found: ID does not exist"
Sep 11 14:38:13.568221 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.568194    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"]
Sep 11 14:38:13.572923 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:13.572903    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8d795dc5-nljg4"]
Sep 11 14:38:14.503601 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:14.503571    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62410c2-a634-4546-957e-30d3b5c30689" path="/var/lib/kubelet/pods/c62410c2-a634-4546-957e-30d3b5c30689/volumes"
Sep 11 14:38:16.169491 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:16.169457    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:38:23.161209 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:23.161173    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pft78"]
Sep 11 14:38:32.420895 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:32.420863    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-ffqqg_f4f5680e-99e4-4e38-9fc2-72f4c674464e/copy-manifests/0.log"
Sep 11 14:38:32.428620 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:32.428576    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-ffqqg_f4f5680e-99e4-4e38-9fc2-72f4c674464e/manager/0.log"
Sep 11 14:38:33.035554 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:33.035525    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7959f48cf9-drqs6_d605701a-788b-44d0-935d-81bd0b75df29/maas-api/0.log"
Sep 11 14:38:33.269527 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:33.269494    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818950-jdpk8_87af1fdb-8d99-4198-8a02-7f2135aaed1d/cleanup/0.log"
Sep 11 14:38:34.457930 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:34.457901    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7vf8d_b454f9b7-f90c-459a-b74b-b8a56d454448/manager/0.log"
Sep 11 14:38:34.582996 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:34.582969    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-frt84_f6a0a2b5-adde-4408-8f6f-aded4c762bf8/kuadrant-console-plugin/0.log"
Sep 11 14:38:34.844050 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:34.843963    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-pft78_0b05281a-cab7-40da-9312-3d9805aaec42/limitador/0.log"
Sep 11 14:38:35.622113 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:35.622081    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-ch7vt_7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e/kube-auth-proxy/0.log"
Sep 11 14:38:36.248992 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:36.248956    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7956fdcb7d-pftq7_d14d4b46-4404-43d0-80b2-1661626901cb/router/0.log"
Sep 11 14:38:36.864873 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:36.864842    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-j7j96_19f8cf4a-b0d8-41a6-aa2f-aee8ef2bd1d2/main/0.log"
Sep 11 14:38:37.226677 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:37.226648    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bx5r7_0eaad85b-7061-4197-a163-7640ab1cdcca/main/0.log"
Sep 11 14:38:37.346612 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:37.346582    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zqgv5_a491f53d-a797-497d-9cf6-ed268b139262/main/0.log"
Sep 11 14:38:42.616543 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:42.616511    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:38:42.697266 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:42.697233    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:38:45.626882 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:45.626852    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qtbqd_04cb311f-1d53-4152-9f5d-66fb2e50ca01/global-pull-secret-syncer/0.log"
Sep 11 14:38:45.739686 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:45.739655    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log"
Sep 11 14:38:45.915087 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:45.914965    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-144.ec2.internal_49ea5c1f950c94bad71aa5473962c9e7/haproxy/0.log"
Sep 11 14:38:46.862507 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:46.862476    2558 scope.go:117] "RemoveContainer" containerID="9aa804b87ecbc3f3088405ff7fe31c5ad4f7c099b1528e930a7fdb4c4fb84133"
Sep 11 14:38:46.870284 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:46.870265    2558 scope.go:117] "RemoveContainer" containerID="d0b7c20503621aa2a390de8c42827e48476d71dd144052c643c31f9cae568798"
Sep 11 14:38:50.740189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:50.740155    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7vf8d_b454f9b7-f90c-459a-b74b-b8a56d454448/manager/0.log"
Sep 11 14:38:50.768610 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:50.768582    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-frt84_f6a0a2b5-adde-4408-8f6f-aded4c762bf8/kuadrant-console-plugin/0.log"
Sep 11 14:38:50.851037 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:50.850996    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-pft78_0b05281a-cab7-40da-9312-3d9805aaec42/limitador/0.log"
Sep 11 14:38:51.498791 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:51.498756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62pnq_902b2429-8e75-40e2-b3fd-b7e1a72ee32b/konnectivity-agent/0.log" limit=1024
Sep 11 14:38:52.872858 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:52.872830    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fcf85d9b5-rfwgf_b415e54f-a579-444d-85eb-a7c268b85bdf/metrics-server/0.log"
Sep 11 14:38:53.167557 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.167527    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blfsk_88ccaba7-8d7b-494a-b37f-8e79f1432218/node-exporter/0.log"
Sep 11 14:38:53.196860 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.196829    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blfsk_88ccaba7-8d7b-494a-b37f-8e79f1432218/kube-rbac-proxy/0.log"
Sep 11 14:38:53.227606 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.227583    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blfsk_88ccaba7-8d7b-494a-b37f-8e79f1432218/init-textfile/0.log"
Sep 11 14:38:53.360996 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.360922    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mgx5s_317fe2d5-83c9-445d-9063-7101718fb92c/kube-rbac-proxy-main/0.log"
Sep 11 14:38:53.384565 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.384533    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mgx5s_317fe2d5-83c9-445d-9063-7101718fb92c/kube-rbac-proxy-self/0.log"
Sep 11 14:38:53.407556 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.407533    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mgx5s_317fe2d5-83c9-445d-9063-7101718fb92c/openshift-state-metrics/0.log"
Sep 11 14:38:53.450968 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.450945    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/prometheus/0.log"
Sep 11 14:38:53.475851 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.475832    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/config-reloader/0.log"
Sep 11 14:38:53.508198 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.508176    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/thanos-sidecar/0.log"
Sep 11 14:38:53.534077 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.534055    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/kube-rbac-proxy-web/0.log"
Sep 11 14:38:53.570154 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.570128    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/kube-rbac-proxy/0.log"
Sep 11 14:38:53.594310 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.594290    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/kube-rbac-proxy-thanos/0.log"
Sep 11 14:38:53.630087 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:53.629985    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_60107236-15aa-40f4-b297-4926b57eec65/init-config-reloader/0.log"
Sep 11 14:38:55.612964 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.612928    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"]
Sep 11 14:38:55.613369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.613276    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62410c2-a634-4546-957e-30d3b5c30689" containerName="maas-api"
Sep 11 14:38:55.613369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.613288    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62410c2-a634-4546-957e-30d3b5c30689" containerName="maas-api"
Sep 11 14:38:55.613369 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.613340    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62410c2-a634-4546-957e-30d3b5c30689" containerName="maas-api"
Sep 11 14:38:55.616204 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.616185    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.619059 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.619038    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ktch4\"/\"default-dockercfg-mxzzh\""
Sep 11 14:38:55.619151 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.619076    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ktch4\"/\"kube-root-ca.crt\""
Sep 11 14:38:55.619497 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.619483    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ktch4\"/\"openshift-service-ca.crt\""
Sep 11 14:38:55.627792 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.627769    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"]
Sep 11 14:38:55.783346 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.783316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-sys\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.783589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.783363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-lib-modules\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.783589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.783382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-proc\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.783589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.783458    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-podres\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.783589 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.783502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchfg\" (UniqueName: \"kubernetes.io/projected/333bc5ac-a241-44a8-b18d-fc587c9db432-kube-api-access-gchfg\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884138    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchfg\" (UniqueName: \"kubernetes.io/projected/333bc5ac-a241-44a8-b18d-fc587c9db432-kube-api-access-gchfg\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884226 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-sys\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-lib-modules\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-proc\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-podres\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-sys\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-proc\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-lib-modules\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.884468 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.884412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/333bc5ac-a241-44a8-b18d-fc587c9db432-podres\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.894533 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.894502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchfg\" (UniqueName: \"kubernetes.io/projected/333bc5ac-a241-44a8-b18d-fc587c9db432-kube-api-access-gchfg\") pod \"perf-node-gather-daemonset-vn62x\" (UID: \"333bc5ac-a241-44a8-b18d-fc587c9db432\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:55.925534 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:55.925510    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:56.200590 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:56.200554    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"]
Sep 11 14:38:56.204103 ip-10-0-140-144 kubenswrapper[2558]: W0911 14:38:56.204072    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod333bc5ac_a241_44a8_b18d_fc587c9db432.slice/crio-c227921dfa010930511855e9c102ec667088ee3105e8ed473bbae720324c50ae WatchSource:0}: Error finding container c227921dfa010930511855e9c102ec667088ee3105e8ed473bbae720324c50ae: Status 404 returned error can't find the container with id c227921dfa010930511855e9c102ec667088ee3105e8ed473bbae720324c50ae
Sep 11 14:38:56.738155 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:56.738118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x" event={"ID":"333bc5ac-a241-44a8-b18d-fc587c9db432","Type":"ContainerStarted","Data":"804c0d02b8a224599619228f1d4c72767ab06c2d09f154bab28307b8ef4a8f3d"}
Sep 11 14:38:56.738582 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:56.738173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x" event={"ID":"333bc5ac-a241-44a8-b18d-fc587c9db432","Type":"ContainerStarted","Data":"c227921dfa010930511855e9c102ec667088ee3105e8ed473bbae720324c50ae"}
Sep 11 14:38:56.738582 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:56.738204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:38:56.763040 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:56.762985    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x" podStartSLOduration=1.762967407 podStartE2EDuration="1.762967407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:56.760223415 +0000 UTC m=+2470.723838387" watchObservedRunningTime="2026-09-11 14:38:56.762967407 +0000 UTC m=+2470.726582379"
Sep 11 14:38:58.488588 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:58.488562    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94z9z_bb62ad6a-225c-4bb0-8421-f79008e5b666/dns/0.log"
Sep 11 14:38:58.516627 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:58.516604    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94z9z_bb62ad6a-225c-4bb0-8421-f79008e5b666/kube-rbac-proxy/0.log"
Sep 11 14:38:58.681580 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:58.681544    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mxh4_c3de9bfd-b0a4-4bc3-802b-d354f5a1a2f2/dns-node-resolver/0.log"
Sep 11 14:38:59.368773 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:59.368737    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7686b6bc78-kcjfj_aba4a02e-1d1b-4d7b-b221-a6a8d29dd083/registry/0.log"
Sep 11 14:38:59.400939 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:38:59.400910    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wjfc_d8eb0a86-1eea-4d55-87a7-1bca0e1ed936/node-ca/0.log"
Sep 11 14:39:00.848355 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:00.848326    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-ch7vt_7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e/kube-auth-proxy/0.log"
Sep 11 14:39:01.040384 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:01.040357    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7956fdcb7d-pftq7_d14d4b46-4404-43d0-80b2-1661626901cb/router/0.log"
Sep 11 14:39:01.991808 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:01.991776    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfdxs_4c93581a-4009-4914-b149-b586d7df7d8f/serve-healthcheck-canary/0.log"
Sep 11 14:39:02.660549 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:02.660521    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c8rm7_fdf104d2-6dd5-4717-ba07-687b56918990/kube-rbac-proxy/0.log"
Sep 11 14:39:02.688045 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:02.688015    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c8rm7_fdf104d2-6dd5-4717-ba07-687b56918990/exporter/0.log"
Sep 11 14:39:02.712390 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:02.712367    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c8rm7_fdf104d2-6dd5-4717-ba07-687b56918990/extractor/0.log"
Sep 11 14:39:02.751578 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:02.751557    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-vn62x"
Sep 11 14:39:05.874189 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:05.874093    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-ffqqg_f4f5680e-99e4-4e38-9fc2-72f4c674464e/manager/0.log"
Sep 11 14:39:05.895153 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:05.895130    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-ffqqg_f4f5680e-99e4-4e38-9fc2-72f4c674464e/copy-manifests/0.log"
Sep 11 14:39:07.196514 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:07.196479    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-ch7vt_7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e/kube-auth-proxy/0.log"
Sep 11 14:39:07.355671 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:07.355639    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7956fdcb7d-pftq7_d14d4b46-4404-43d0-80b2-1661626901cb/router/0.log"
Sep 11 14:39:07.394966 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:07.394890    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-779774756-jc4c7_b52c9317-49c4-4dc0-80b9-5437ed074ed2/manager/0.log"
Sep 11 14:39:09.174693 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:09.174665    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-ch7vt_7804bec9-7ec2-43d4-b7d4-4f9e40a6ed7e/kube-auth-proxy/0.log"
Sep 11 14:39:09.329577 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:09.329548    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7956fdcb7d-pftq7_d14d4b46-4404-43d0-80b2-1661626901cb/router/0.log"
Sep 11 14:39:12.634975 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:12.634941    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7fdsd_4d2dc9d3-1a28-4e10-a355-8a22134d9cc9/migrator/0.log"
Sep 11 14:39:12.659391 ip-10-0-140-144 kubenswrapper[2558]: I0911 14:39:12.659364    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7fdsd_4d2dc9d3-1a28-4e10-a355-8a22134d9cc9/graceful-termination/0.log"