Sep 12 00:14:29.929542 ip-10-0-140-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 12 00:14:29.929552 ip-10-0-140-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 12 00:14:29.929559 ip-10-0-140-187 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 12 00:14:29.929767 ip-10-0-140-187 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 12 00:14:39.928886 ip-10-0-140-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 12 00:14:39.928903 ip-10-0-140-187 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2f6a5e0fecda4daeac22e2343bbc0155 --
Sep 12 00:16:36.948751 ip-10-0-140-187 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 00:16:37.324354 ip-10-0-140-187 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:16:37.324354 ip-10-0-140-187 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 00:16:37.324354 ip-10-0-140-187 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:16:37.324354 ip-10-0-140-187 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 00:16:37.324354 ip-10-0-140-187 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:16:37.326094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.326008    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 00:16:37.329449 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329433    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329450    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329454    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329457    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329460    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329464    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329467    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329480    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329483    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329497    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329500    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329504    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329507    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329509    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:16:37.329496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329512    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329515    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329518    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329524    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329527    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329529    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329532    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329535    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329537    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329540    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329542    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329545    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329548    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329550    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329553    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329556    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329558    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329561    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329563    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:16:37.329877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329566    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329568    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329570    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329573    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329576    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329578    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329580    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329583    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329585    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329588    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329591    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329594    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329596    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329598    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329601    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329604    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329607    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329611    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329616    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329619    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:16:37.330337 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329622    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329625    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329627    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329630    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329633    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329636    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329638    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329641    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329644    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329646    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329649    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329651    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329654    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329656    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329659    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329662    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329664    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329667    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329669    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329672    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:16:37.330836 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329675    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329679    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329683    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329686    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329689    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329692    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329695    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329699    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329702    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329704    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329707    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329710    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329712    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329715    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329719    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329722    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329726    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329728    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329731    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:16:37.331304 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329734    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329737    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329740    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329743    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329746    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329749    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.329751    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330431    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330438    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330440    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330444    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330446    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330449    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330452    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330454    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330457    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330460    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330463    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330466    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330482    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:16:37.331784 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330486    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330489    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330491    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330494    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330497    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330500    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330502    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330505    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330507    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330510    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330513    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330515    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330518    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330521    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330523    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330526    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330529    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330531    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330534    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330536    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:16:37.332310 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330539    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330541    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330544    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330547    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330550    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330555    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330559    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330562    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330565    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330568    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330571    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330574    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330577    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330579    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330582    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330585    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330587    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330589    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330592    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330595    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:16:37.332808 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330597    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330600    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330602    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330605    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330608    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330610    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330613    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330615    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330618    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330620    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330623    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330625    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330628    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330630    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330633    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330636    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330638    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330641    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330643    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330646    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:16:37.333375 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330649    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330652    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330655    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330657    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330660    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330663    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330665    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330668    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330672    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330674    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330677    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330679    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330683    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330686    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330689    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330691    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330694    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330700    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330703    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:16:37.333877 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330706    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330709    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330713    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330716    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330718    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330722    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.330724    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331080    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331086    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331090    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331093    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331096    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331099    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331101    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331104    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331106    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331109    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331114    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331117    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:16:37.334355 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331119    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331122    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331124    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331127    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331130    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331132    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331135    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331137    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331140    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331143    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331145    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331149    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331151    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331154    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331157    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331159    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331162    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331164    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331167    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331170    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:16:37.334822 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331173    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331175    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331178    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331180    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331183    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331187    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331190    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331192    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331195    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331198    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331202    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331204    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331207    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331209    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331213    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331216    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331219    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331222    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331224    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:16:37.335308 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331227    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331230    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331232    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331235    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331239    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331241    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331244    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331246    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331249    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331251    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331254    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331256    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331259    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331261    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331264    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331266    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331268    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331271    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331275    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331277    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:16:37.335790 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331279    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331282    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331284    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331288    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331293    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331296    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331298    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331301    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331303    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331306    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331308    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331311    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331313    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331316    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331319    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331321    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331325    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331327    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331330    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:16:37.336275 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331332    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331334    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331337    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331339    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331342    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331344    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331347    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331349    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.331351    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331420    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331461    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331480    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331485    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331488    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331492    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331496    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331500    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331503    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331507    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331510    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331514    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331517    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 00:16:37.336750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331519    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331522    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331525    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331528    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331531    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331534    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331537    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331540    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331544    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331547    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331551    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331554    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331557    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331560    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331563    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331566    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331569    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331573    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331575    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331579    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331582    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331585    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331588    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331591    2562 flags.go:64] FLAG: --enable-server="true"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331593    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 00:16:37.337273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331598    2562 flags.go:64] FLAG: --event-burst="100"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331601    2562 flags.go:64] FLAG: --event-qps="50"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331604    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331607    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331611    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331614    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331617    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331621    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331624    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331627    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331629    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331632    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331635    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331638    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331641    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331645    2562 flags.go:64] FLAG: --feature-gates=""
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331649    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331651    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331655    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331658    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331661    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331664    2562 flags.go:64] FLAG: --help="false"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331666    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331669    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 00:16:37.337961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331673    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331675    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331679    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331682    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331685    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331688    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331690    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331693    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331696    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331699    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331702    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331705    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331708    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331712    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331715    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331717    2562 flags.go:64] FLAG: --lock-file=""
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331720    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331723    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331726    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331732    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331735    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331737    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331740    2562 flags.go:64] FLAG: --logging-format="text"
Sep 12 00:16:37.338937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331743    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331746    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331749    2562 flags.go:64] FLAG: --manifest-url=""
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331752    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331756    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331759    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331763    2562 flags.go:64] FLAG: --max-pods="110"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331766    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331769    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331772    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331775    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331778    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331780    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331783    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331790    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331793    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331797    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331800    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331803    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331808    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331811    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331814    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331817    2562 flags.go:64] FLAG: --port="10250"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331820    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 00:16:37.339648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331823    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bbffea4897fd09c2"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331826    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331829    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331832    2562 flags.go:64] FLAG: --register-node="true"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331835    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331838    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331841    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331844    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331849    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331852    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331856    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331859    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331862    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331865    2562 flags.go:64] FLAG: --runonce="false"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331868    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331871    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331874    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331877    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331880    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331883    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331886    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331890    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331893    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331895    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331898    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331901    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 00:16:37.340234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331906    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331909    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331912    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331918    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331921    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331923    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331928    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331931    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331934    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331937    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331940    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331943    2562 flags.go:64] FLAG: --v="2"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331947    2562 flags.go:64] FLAG: --version="false"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331951    2562 flags.go:64] FLAG: --vmodule=""
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331956    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.331959    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332071    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332074    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332078    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332081    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332084    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332088    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332091    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:16:37.340872 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332094    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332097    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332100    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332103    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332106    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332108    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332111    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332114    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332116    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332120    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332123    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332125    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332128    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332130    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332133    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332136    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332139    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332141    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332144    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332147    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:16:37.341428 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332150    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332152    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332155    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332159    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332162    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332164    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332167    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332170    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332172    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332175    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332178    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332180    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332183    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332186    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332188    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332191    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332194    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332197    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332199    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332202    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:16:37.341979 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332205    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332208    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332211    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332213    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332216    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332219    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332221    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332224    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332227    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332230    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332232    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332235    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332238    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332240    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332243    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332248    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332251    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332253    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332257    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:16:37.342505 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332260    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332264    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332267    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332269    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332272    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332275    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332278    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332281    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332283    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332286    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332289    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332291    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332294    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332297    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332301    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332304    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332306    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332310    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332314    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:16:37.342973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332317    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332320    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332323    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332327    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332330    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332333    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332337    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332340    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332343    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332348    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332350    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332353    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332356    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.332359    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.332365    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:16:37.343437 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.340855    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.340870    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.340896    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.340903    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340977    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340982    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340986    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340989    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340993    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340995    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.340998    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341001    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341004    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341006    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341009    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341011    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341014    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341018    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341022    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:16:37.343928 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341025    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341028    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341030    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341033    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341036    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341039    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341041    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341045    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341049    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341053    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341056    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341059    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341061    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341064    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341067    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341070    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341073    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341076    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341078    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341081    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:16:37.344406 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341084    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341087    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341089    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341092    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341095    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341098    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341100    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341103    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341105    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341108    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341111    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341113    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341116    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341120    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341124    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341127    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341130    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341133    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341135    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341138    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:16:37.344913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341141    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341144    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341146    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341149    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341152    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341154    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341157    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341160    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341163    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341165    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341169    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341171    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341174    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341177    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341180    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341182    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341185    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341188    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341190    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341193    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:16:37.345433 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341195    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341198    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341201    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341203    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341207    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341209    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341212    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341214    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341217    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341219    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341222    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341224    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341227    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341229    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341232    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341234    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341237    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341240    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341243    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341245    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:16:37.345923 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341248    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341250    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341253    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341256    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.341260    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341372    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341377    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341380    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341383    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341386    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341388    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341391    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341393    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341396    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341399    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:16:37.346423 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341401    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341404    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341408    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341410    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341413    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341416    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341419    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341421    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341425    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341429    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341432    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341435    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341438    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341441    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341444    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341446    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341449    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341451    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341454    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:16:37.346812 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341457    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341459    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341462    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341464    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341467    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341487    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341490    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341493    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341496    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341498    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341501    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341504    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341507    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341509    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341512    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341514    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341518    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341521    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341525    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:16:37.347267 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341528    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341531    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341534    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341537    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341540    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341543    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341545    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341548    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341550    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341553    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341555    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341558    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341560    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341563    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341566    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341568    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341571    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341573    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341576    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341578    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:16:37.347794 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341581    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341583    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341586    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341589    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341591    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341594    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341597    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341600    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341603    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341605    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341608    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341611    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341614    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341616    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341619    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341621    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341624    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341626    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341629    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:16:37.348301 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341633    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341636    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341639    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341642    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341645    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341648    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341650    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341653    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341656    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341659    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341661    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:37.341664    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.341670    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.341769    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.343792    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 00:16:37.348780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.344730    2562 server.go:1019] "Starting client certificate rotation"
Sep 12 00:16:37.349163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.344824    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:16:37.349163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.345647    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:16:37.349163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.348009    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:16:37.350828 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.350808    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:16:37.365030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.365009    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 12 00:16:37.369639 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.369620    2562 log.go:25] "Validated CRI v1 image API"
Sep 12 00:16:37.370396 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.370381    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 00:16:37.373240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.373222    2562 fs.go:135] Filesystem UUIDs: map[21001d73-91f1-4af7-b12d-98d4b5bea89f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c8ccd3ce-747d-4821-b890-07e1cece3ce0:/dev/nvme0n1p4]
Sep 12 00:16:37.373299 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.373238    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 00:16:37.380408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.380297    2562 manager.go:216] Machine: {Timestamp:2026-09-12 00:16:37.379018329 +0000 UTC m=+0.316324839 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103132 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269a81c9f8dd2cfa8acd86cf375f4e SystemUUID:ec269a81-c9f8-dd2c-fa8a-cd86cf375f4e BootID:2f6a5e0f-ecda-4dae-ac22-e2343bbc0155 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:03:9b:b3:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:03:9b:b3:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:b0:52:9b:ad:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 00:16:37.380408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.380399    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 00:16:37.380554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.380489    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 00:16:37.380929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.380913    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:16:37.383643 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.383620    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 00:16:37.383775 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.383646    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 00:16:37.384356 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.384347    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 00:16:37.384385 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.384358    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 00:16:37.384385 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.384367    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:16:37.385083 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.385072    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:16:37.385753 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.385745    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 00:16:37.388367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388357    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:16:37.388466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388457    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 00:16:37.388514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388500    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 00:16:37.388550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388518    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 00:16:37.388550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388536    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 00:16:37.388550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388545    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 12 00:16:37.388669 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388556    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 00:16:37.388887 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388876    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:16:37.388929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.388894    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:16:37.390750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.390730    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 00:16:37.392810 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.392796    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 00:16:37.392871 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.392825    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 00:16:37.394401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394389    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 00:16:37.394442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394405    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 00:16:37.394442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394411    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 00:16:37.394442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394417    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 00:16:37.394442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394425    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 00:16:37.394442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394430    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 00:16:37.394442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394444    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 00:16:37.394620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394450    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 00:16:37.394620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394457    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 00:16:37.394620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394462    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 00:16:37.394620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394484    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 00:16:37.394620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.394490    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 00:16:37.395801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.395787    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 00:16:37.395801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.395797    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 00:16:37.398728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.398699    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 00:16:37.398798 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.398736    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 00:16:37.398866 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.398787    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 00:16:37.399554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.399540    2562 server.go:1268] "Started kubelet"
Sep 12 00:16:37.399636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.399616    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 00:16:37.399702 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.399676    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 00:16:37.399743 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.399722    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 00:16:37.400955 ip-10-0-140-187 systemd[1]: Started Kubernetes Kubelet.
Sep 12 00:16:37.401273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.401261    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 00:16:37.402576 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.402562    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 00:16:37.403597 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.403571    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 00:16:37.405483 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.404675    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-187.ec2.internal.18d46aa25400c228  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-187.ec2.internal,UID:ip-10-0-140-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-187.ec2.internal,},FirstTimestamp:2026-09-12 00:16:37.399511592 +0000 UTC m=+0.336818104,LastTimestamp:2026-09-12 00:16:37.399511592 +0000 UTC m=+0.336818104,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-187.ec2.internal,}"
Sep 12 00:16:37.406095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.406076    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 00:16:37.406844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.406830    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 00:16:37.406964 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.406942    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.407653 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407631    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 00:16:37.407788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407661    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 00:16:37.407788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407678    2562 factory.go:55] Registering systemd factory
Sep 12 00:16:37.407788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407692    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 12 00:16:37.407788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407631    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 00:16:37.407788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407758    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 00:16:37.407788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407765    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 00:16:37.408092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407926    2562 factory.go:153] Registering CRI-O factory
Sep 12 00:16:37.408092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407936    2562 factory.go:223] Registration of the crio container factory successfully
Sep 12 00:16:37.408092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.407987    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 00:16:37.408092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.408018    2562 factory.go:103] Registering Raw factory
Sep 12 00:16:37.408092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.408028    2562 manager.go:1196] Started watching for new ooms in manager
Sep 12 00:16:37.409048 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.409025    2562 manager.go:349] Starting recovery of all containers
Sep 12 00:16:37.418898 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.418873    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 12 00:16:37.418999 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.418925    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 12 00:16:37.420650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.420633    2562 manager.go:354] Recovery completed
Sep 12 00:16:37.422161 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.422143    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-555qt"
Sep 12 00:16:37.425722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.425709    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.427188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427172    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.427258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427197    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.427258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427207    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.427789 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427768    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 00:16:37.427789 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427786    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 00:16:37.427932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427777    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.427932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.427802    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:16:37.428914 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.428899    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-555qt"
Sep 12 00:16:37.429695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.429679    2562 policy_none.go:49] "None policy: Start"
Sep 12 00:16:37.429695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.429695    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 00:16:37.429853 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.429707    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 00:16:37.429953 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.429937    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.430019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.429968    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.430019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.429983    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.430820 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.430721    2562 policy_none.go:47] "Start"
Sep 12 00:16:37.468106 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468088    2562 manager.go:337] "Starting Device Plugin manager"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.468130    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468142    2562 server.go:87] "Starting device plugin registration server"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468426    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468438    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468692    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468780    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.468789    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.469093    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 00:16:37.481739 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.469162    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.562287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.562259    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 00:16:37.563482 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.563457    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 00:16:37.563545 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.563519    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 00:16:37.563545 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.563541    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 00:16:37.563640 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.563626    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 12 00:16:37.566440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.566418    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:16:37.569089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.569077    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.570161 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.570140    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.570244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.570173    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.570244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.570189    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.570244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.570210    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.580218 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.580167    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.580218 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.580186    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-187.ec2.internal\": node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.634047 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.634023    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.664504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.664483    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"]
Sep 12 00:16:37.664590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.664557    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.665336 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.665322    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.665412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.665353    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.665412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.665370    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.667675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.667661    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.667794 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.667777    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.667842 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.667811    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.668343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.668326    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.668420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.668356    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.668420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.668331    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.668420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.668371    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.668420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.668386    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.668420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.668400    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.670552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.670537    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.670641 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.670561    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:16:37.671190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.671177    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:16:37.671259 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.671201    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:16:37.671259 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.671216    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:16:37.683998 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.683979    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-187.ec2.internal\" not found" node="ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.688460 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.688445    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-187.ec2.internal\" not found" node="ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.734537 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.734517    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.809156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.809132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/188d48a6c12b055e11435525c2a8acb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal\" (UID: \"188d48a6c12b055e11435525c2a8acb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.809156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.809156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188d48a6c12b055e11435525c2a8acb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal\" (UID: \"188d48a6c12b055e11435525c2a8acb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.809287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.809173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f8f2c475d91043fb3907657c68af0ed-config\") pod \"kube-apiserver-proxy-ip-10-0-140-187.ec2.internal\" (UID: \"7f8f2c475d91043fb3907657c68af0ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.835589 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.835537    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.909775 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.909750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/188d48a6c12b055e11435525c2a8acb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal\" (UID: \"188d48a6c12b055e11435525c2a8acb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.909877 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.909776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188d48a6c12b055e11435525c2a8acb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal\" (UID: \"188d48a6c12b055e11435525c2a8acb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.909877 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.909793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f8f2c475d91043fb3907657c68af0ed-config\") pod \"kube-apiserver-proxy-ip-10-0-140-187.ec2.internal\" (UID: \"7f8f2c475d91043fb3907657c68af0ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.909877 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.909825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f8f2c475d91043fb3907657c68af0ed-config\") pod \"kube-apiserver-proxy-ip-10-0-140-187.ec2.internal\" (UID: \"7f8f2c475d91043fb3907657c68af0ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.909877 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.909836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/188d48a6c12b055e11435525c2a8acb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal\" (UID: \"188d48a6c12b055e11435525c2a8acb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.909877 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.909849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188d48a6c12b055e11435525c2a8acb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal\" (UID: \"188d48a6c12b055e11435525c2a8acb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.935853 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:37.935833    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:37.987004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.986976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:37.990613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:37.990595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:38.036029 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.036007    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.137833 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.137797    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.239314 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.239285    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.340555 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.340521    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.344681 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.344660    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 00:16:38.344811 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.344794    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:16:38.432194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.432118    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 00:11:38 +0000 UTC" deadline="2028-03-11 07:57:41.110960022 +0000 UTC"
Sep 12 00:16:38.432194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.432152    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13111h41m2.678811983s"
Sep 12 00:16:38.441138 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.441122    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.540251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.540215    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:16:38.542269 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.542249    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.643428 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.643398    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.694213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.694135    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:16:38.716105 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:38.716063    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188d48a6c12b055e11435525c2a8acb7.slice/crio-7d3e11af15c9d829db9bb32e48b070a1b88ab3ec271b9e09fa60f326f174c53d WatchSource:0}: Error finding container 7d3e11af15c9d829db9bb32e48b070a1b88ab3ec271b9e09fa60f326f174c53d: Status 404 returned error can't find the container with id 7d3e11af15c9d829db9bb32e48b070a1b88ab3ec271b9e09fa60f326f174c53d
Sep 12 00:16:38.716486 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:38.716469    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8f2c475d91043fb3907657c68af0ed.slice/crio-50f7293a0739c01fb2f941f5512cae4233ce744de64f1b27c5c78b145eb82a5d WatchSource:0}: Error finding container 50f7293a0739c01fb2f941f5512cae4233ce744de64f1b27c5c78b145eb82a5d: Status 404 returned error can't find the container with id 50f7293a0739c01fb2f941f5512cae4233ce744de64f1b27c5c78b145eb82a5d
Sep 12 00:16:38.720086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.720070    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:16:38.743572 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:38.743547    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-187.ec2.internal\" not found"
Sep 12 00:16:38.794345 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.794320    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:16:38.807554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.807527    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:38.822311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.822292    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:16:38.823645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.823631    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal"
Sep 12 00:16:38.834748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:38.834729    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:16:39.356195 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.356166    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:16:39.389359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.389323    2562 apiserver.go:52] "Watching apiserver"
Sep 12 00:16:39.398640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.398608    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 00:16:39.399184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.399157    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-7x4r8","openshift-ovn-kubernetes/ovnkube-node-7wtr4","kube-system/global-pull-secret-syncer-7tm66","kube-system/konnectivity-agent-2rthv","kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nw9lf","openshift-dns/node-resolver-ttcd9","openshift-image-registry/node-ca-gxj9b","openshift-multus/multus-additional-cni-plugins-mrzr4","openshift-multus/multus-sw4wh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal","openshift-multus/network-metrics-daemon-qvk2b","openshift-network-diagnostics/network-check-target-vdzl6"]
Sep 12 00:16:39.402230 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.402205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.404665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.404638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.407033 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.406960    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 00:16:39.407838 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.407806    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.408056 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.408035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.408351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.408315    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tp4kk\""
Sep 12 00:16:39.408580 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.408563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.408657 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.408631    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:39.410742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.409935    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b7wkj\""
Sep 12 00:16:39.410742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.410142    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 00:16:39.410742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.410348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.410742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.410580    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.411358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.411334    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.415934 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.415900    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 00:16:39.416035 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.415959    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 00:16:39.416659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.416181    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:39.416659 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.416241    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:39.416659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.416188    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-djnt9\""
Sep 12 00:16:39.416659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.416329    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkbb\" (UniqueName: \"kubernetes.io/projected/a1a021a5-f8de-48be-8456-1c71ef09a626-kube-api-access-rqkbb\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-socket-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v74k\" (UniqueName: \"kubernetes.io/projected/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-kube-api-access-7v74k\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be73927c-b191-4bef-8ded-c6bee22f4ae0-kubelet-config\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce54e557-520d-4218-b42b-2571f6e04a5b-agent-certs\") pod \"konnectivity-agent-2rthv\" (UID: \"ce54e557-520d-4218-b42b-2571f6e04a5b\") " pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-sys-fs\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a021a5-f8de-48be-8456-1c71ef09a626-tmp-dir\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-device-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be73927c-b191-4bef-8ded-c6bee22f4ae0-dbus\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce54e557-520d-4218-b42b-2571f6e04a5b-konnectivity-ca\") pod \"konnectivity-agent-2rthv\" (UID: \"ce54e557-520d-4218-b42b-2571f6e04a5b\") " pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.417994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1a021a5-f8de-48be-8456-1c71ef09a626-hosts-file\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.418227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.418027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-registration-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.420322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.419856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.423800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.420724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jg8qq\""
Sep 12 00:16:39.423800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.421676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:39.423800 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.421747    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:39.424550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.424102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.424550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.424500    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.425146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.425123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.425227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.425174    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 00:16:39.425227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.425194    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.425227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.425129    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.425343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.425229    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-th2hh\""
Sep 12 00:16:39.427647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.426994    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 00:16:39.427647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.427230    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 00:16:39.428804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.428783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.432362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.432342    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 00:16:39.432526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.432505    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 00:16:39.432631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.432540    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 00:16:39.432823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.432796    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dmmn8\""
Sep 12 00:16:39.432899 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.432400    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 00:16:39.432959 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.432393    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.434229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.433122    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.434229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.433328    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 00:16:39.434229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.433632    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:16:39.434454 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.434270    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.434454 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.434337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.435984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.435965    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q69j2\""
Sep 12 00:16:39.437685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.437626    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ltmqg\""
Sep 12 00:16:39.437781 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.437695    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 00:16:39.440992 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.440961    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.441108 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.441016    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 00:16:39.441108 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.441052    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.441409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.441386    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:16:39.441633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.441614    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8zxtt\""
Sep 12 00:16:39.441723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.441707    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 00:16:39.509038 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.508871    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 00:16:39.518949 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.518920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a021a5-f8de-48be-8456-1c71ef09a626-tmp-dir\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.519066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.518969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-run\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.519066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.518996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-lib-modules\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.519066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-tuned\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.519066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-multus-certs\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce54e557-520d-4218-b42b-2571f6e04a5b-konnectivity-ca\") pod \"konnectivity-agent-2rthv\" (UID: \"ce54e557-520d-4218-b42b-2571f6e04a5b\") " pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-var-lib-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-cnibin\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-k8s-cni-cncf-io\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-socket-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.519271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v74k\" (UniqueName: \"kubernetes.io/projected/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-kube-api-access-7v74k\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519288    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-cni-bin\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-cni-netd\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-cni-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be73927c-b191-4bef-8ded-c6bee22f4ae0-kubelet-config\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-cni-binary-copy\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-hostroot\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-device-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-modprobe-d\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysctl-d\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-etc-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4wd\" (UniqueName: \"kubernetes.io/projected/936b2277-3619-42f3-aede-3c77c9e6285c-kube-api-access-mv4wd\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.519687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-netns\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1a021a5-f8de-48be-8456-1c71ef09a626-hosts-file\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-env-overrides\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519780    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysconfig\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-sys\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-host\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-kubelet\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-ovn\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.519962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-kubelet\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-systemd\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-systemd-units\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-socket-dir-parent\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-var-lib-kubelet\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/936b2277-3619-42f3-aede-3c77c9e6285c-iptables-alerter-script\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.520409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a021a5-f8de-48be-8456-1c71ef09a626-tmp-dir\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47vt\" (UniqueName: \"kubernetes.io/projected/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-kube-api-access-t47vt\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-cni-multus\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-registration-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-host\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be73927c-b191-4bef-8ded-c6bee22f4ae0-kubelet-config\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1a021a5-f8de-48be-8456-1c71ef09a626-hosts-file\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwznk\" (UniqueName: \"kubernetes.io/projected/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-kube-api-access-jwznk\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-registration-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-socket-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-device-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cnibin\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkbb\" (UniqueName: \"kubernetes.io/projected/a1a021a5-f8de-48be-8456-1c71ef09a626-kube-api-access-rqkbb\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4sl\" (UniqueName: \"kubernetes.io/projected/e5b87268-b568-4b87-b832-4ff5a4d65a95-kube-api-access-xv4sl\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.521253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-serviceca\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.520994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce54e557-520d-4218-b42b-2571f6e04a5b-agent-certs\") pod \"konnectivity-agent-2rthv\" (UID: \"ce54e557-520d-4218-b42b-2571f6e04a5b\") " pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-sys-fs\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-slash\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-node-log\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-kubernetes\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce54e557-520d-4218-b42b-2571f6e04a5b-konnectivity-ca\") pod \"konnectivity-agent-2rthv\" (UID: \"ce54e557-520d-4218-b42b-2571f6e04a5b\") " pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysctl-conf\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkp5f\" (UniqueName: \"kubernetes.io/projected/8a1cc4ee-f316-444d-af21-4fa2788a9a28-kube-api-access-mkp5f\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-systemd\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-log-socket\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/936b2277-3619-42f3-aede-3c77c9e6285c-host-slash\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7tr\" (UniqueName: \"kubernetes.io/projected/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-kube-api-access-kn7tr\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-sys-fs\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.522190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be73927c-b191-4bef-8ded-c6bee22f4ae0-dbus\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-run-netns\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovnkube-config\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovnkube-script-lib\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be73927c-b191-4bef-8ded-c6bee22f4ae0-dbus\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-conf-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521647    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k679\" (UniqueName: \"kubernetes.io/projected/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-kube-api-access-4k679\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-system-cni-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-cni-bin\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-daemon-config\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-etc-kubernetes\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a1cc4ee-f316-444d-af21-4fa2788a9a28-tmp\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.521938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovn-node-metrics-cert\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.522007    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:39.522884 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.522145    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:40.022121636 +0000 UTC m=+2.959428159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:39.525679 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.525658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce54e557-520d-4218-b42b-2571f6e04a5b-agent-certs\") pod \"konnectivity-agent-2rthv\" (UID: \"ce54e557-520d-4218-b42b-2571f6e04a5b\") " pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.536280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.535825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkbb\" (UniqueName: \"kubernetes.io/projected/a1a021a5-f8de-48be-8456-1c71ef09a626-kube-api-access-rqkbb\") pod \"node-resolver-ttcd9\" (UID: \"a1a021a5-f8de-48be-8456-1c71ef09a626\") " pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.536280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.536228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v74k\" (UniqueName: \"kubernetes.io/projected/8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd-kube-api-access-7v74k\") pod \"aws-ebs-csi-driver-node-xqw77\" (UID: \"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.568363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.568298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal" event={"ID":"7f8f2c475d91043fb3907657c68af0ed","Type":"ContainerStarted","Data":"50f7293a0739c01fb2f941f5512cae4233ce744de64f1b27c5c78b145eb82a5d"}
Sep 12 00:16:39.569852 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.569824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal" event={"ID":"188d48a6c12b055e11435525c2a8acb7","Type":"ContainerStarted","Data":"7d3e11af15c9d829db9bb32e48b070a1b88ab3ec271b9e09fa60f326f174c53d"}
Sep 12 00:16:39.589769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.589741    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8dvn"
Sep 12 00:16:39.604127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.604097    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8dvn"
Sep 12 00:16:39.623063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.622989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-var-lib-kubelet\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.623063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/936b2277-3619-42f3-aede-3c77c9e6285c-iptables-alerter-script\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.623063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47vt\" (UniqueName: \"kubernetes.io/projected/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-kube-api-access-t47vt\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-cni-multus\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-var-lib-kubelet\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-host\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-host\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwznk\" (UniqueName: \"kubernetes.io/projected/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-kube-api-access-jwznk\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cnibin\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-cni-multus\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.623311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4sl\" (UniqueName: \"kubernetes.io/projected/e5b87268-b568-4b87-b832-4ff5a4d65a95-kube-api-access-xv4sl\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-serviceca\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-slash\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-node-log\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-kubernetes\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysctl-conf\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkp5f\" (UniqueName: \"kubernetes.io/projected/8a1cc4ee-f316-444d-af21-4fa2788a9a28-kube-api-access-mkp5f\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-systemd\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-log-socket\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/936b2277-3619-42f3-aede-3c77c9e6285c-host-slash\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7tr\" (UniqueName: \"kubernetes.io/projected/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-kube-api-access-kn7tr\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-run-netns\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovnkube-config\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovnkube-script-lib\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.623791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/936b2277-3619-42f3-aede-3c77c9e6285c-iptables-alerter-script\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-conf-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-conf-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k679\" (UniqueName: \"kubernetes.io/projected/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-kube-api-access-4k679\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/936b2277-3619-42f3-aede-3c77c9e6285c-host-slash\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-system-cni-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-cni-bin\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-daemon-config\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysctl-conf\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-etc-kubernetes\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.623994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a1cc4ee-f316-444d-af21-4fa2788a9a28-tmp\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovn-node-metrics-cert\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-run-netns\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-run\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-lib-modules\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-tuned\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.624735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-multus-certs\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-var-lib-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-slash\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-cnibin\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-node-log\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-k8s-cni-cncf-io\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-cni-bin\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-system-cni-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-cni-netd\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-cni-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-cni-binary-copy\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-hostroot\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-modprobe-d\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysctl-d\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.625714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-etc-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4wd\" (UniqueName: \"kubernetes.io/projected/936b2277-3619-42f3-aede-3c77c9e6285c-kube-api-access-mv4wd\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-daemon-config\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-netns\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-cni-bin\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-env-overrides\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-cnibin\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cnibin\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysconfig\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-sys\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-host\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-kubelet\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-ovn\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-kubelet\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-systemd\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-systemd-units\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.626581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-systemd-units\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-serviceca\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysconfig\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-env-overrides\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-var-lib-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-sys\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.624182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-kubernetes\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-k8s-cni-cncf-io\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-host\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-systemd\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-multus-certs\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-log-socket\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-run-ovn\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-modprobe-d\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-cni-bin\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-systemd\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-var-lib-kubelet\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.627402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-cni-netd\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-cni-dir\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625522    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-run-ovn-kubernetes\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-hostroot\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-kubelet\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625727    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-etc-openvswitch\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5b87268-b568-4b87-b832-4ff5a4d65a95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-sysctl-d\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-host-run-netns\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovnkube-script-lib\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-lib-modules\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-cni-binary-copy\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a1cc4ee-f316-444d-af21-4fa2788a9a28-run\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.625971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-etc-kubernetes\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.626001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:39.628329 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.627218    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.627302    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:40.127273502 +0000 UTC m=+3.064580024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.627580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.626042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-socket-dir-parent\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.627791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-multus-socket-dir-parent\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.627980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovnkube-config\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.628002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a1cc4ee-f316-444d-af21-4fa2788a9a28-etc-tuned\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.629114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.628774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5b87268-b568-4b87-b832-4ff5a4d65a95-ovn-node-metrics-cert\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.631584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.631518    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a1cc4ee-f316-444d-af21-4fa2788a9a28-tmp\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.651059 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.650989    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:16:39.651059 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.651015    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:16:39.651059 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.651029    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:39.651310 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:39.651111    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:40.151089404 +0000 UTC m=+3.088395923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:39.653082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.653055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47vt\" (UniqueName: \"kubernetes.io/projected/592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d-kube-api-access-t47vt\") pod \"multus-additional-cni-plugins-mrzr4\" (UID: \"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d\") " pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.653082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.653073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwznk\" (UniqueName: \"kubernetes.io/projected/4353938a-ab7e-4fd7-b93c-2336a53ff5f9-kube-api-access-jwznk\") pod \"node-ca-gxj9b\" (UID: \"4353938a-ab7e-4fd7-b93c-2336a53ff5f9\") " pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:39.653306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.653279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7tr\" (UniqueName: \"kubernetes.io/projected/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-kube-api-access-kn7tr\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:39.654020 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.653968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4sl\" (UniqueName: \"kubernetes.io/projected/e5b87268-b568-4b87-b832-4ff5a4d65a95-kube-api-access-xv4sl\") pod \"ovnkube-node-7wtr4\" (UID: \"e5b87268-b568-4b87-b832-4ff5a4d65a95\") " pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.654686 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.654665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4wd\" (UniqueName: \"kubernetes.io/projected/936b2277-3619-42f3-aede-3c77c9e6285c-kube-api-access-mv4wd\") pod \"iptables-alerter-7x4r8\" (UID: \"936b2277-3619-42f3-aede-3c77c9e6285c\") " pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.656022 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.656004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k679\" (UniqueName: \"kubernetes.io/projected/4ae99fad-f382-45ba-a89a-8cc391fb3d9f-kube-api-access-4k679\") pod \"multus-sw4wh\" (UID: \"4ae99fad-f382-45ba-a89a-8cc391fb3d9f\") " pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.656112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.656036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkp5f\" (UniqueName: \"kubernetes.io/projected/8a1cc4ee-f316-444d-af21-4fa2788a9a28-kube-api-access-mkp5f\") pod \"tuned-nw9lf\" (UID: \"8a1cc4ee-f316-444d-af21-4fa2788a9a28\") " pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.725241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.725208    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ttcd9"
Sep 12 00:16:39.744406 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.744380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77"
Sep 12 00:16:39.755298 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.755260    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:16:39.760996 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.760977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nw9lf"
Sep 12 00:16:39.769723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.769700    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mrzr4"
Sep 12 00:16:39.779650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.779626    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:16:39.788717 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.788693    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sw4wh"
Sep 12 00:16:39.803567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.803544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7x4r8"
Sep 12 00:16:39.810566 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:39.810546    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gxj9b"
Sep 12 00:16:40.030506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:40.030398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:40.030682 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.030556    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:40.030682 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.030643    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:41.030622945 +0000 UTC m=+3.967929445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:40.131397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:40.131363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:40.131601 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.131492    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:40.131601 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.131581    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:41.131560157 +0000 UTC m=+4.068866671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:40.232340 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:40.232304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:40.232533 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.232497    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:16:40.232533 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.232522    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:16:40.232671 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.232536    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:40.232671 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.232593    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:41.232577807 +0000 UTC m=+4.169884306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:40.564742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:40.564702    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:40.565176 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:40.564842    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:40.608001 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:40.607958    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:11:40 +0000 UTC" deadline="2028-04-27 21:56:53.812948811 +0000 UTC"
Sep 12 00:16:40.608001 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:40.607997    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14253h40m13.204956208s"
Sep 12 00:16:40.817001 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.816971    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4353938a_ab7e_4fd7_b93c_2336a53ff5f9.slice/crio-9c65fab870d14059b6604ac20a78d2a965285b8f6a24d09bc58beaeb337325a5 WatchSource:0}: Error finding container 9c65fab870d14059b6604ac20a78d2a965285b8f6a24d09bc58beaeb337325a5: Status 404 returned error can't find the container with id 9c65fab870d14059b6604ac20a78d2a965285b8f6a24d09bc58beaeb337325a5
Sep 12 00:16:40.818389 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.818368    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7191b5_a8fd_46a8_b20a_bdf7e3a84efd.slice/crio-3327555be0e2169e6e626b8eed281bf31d3b2fa28c99b11152034ed90472005b WatchSource:0}: Error finding container 3327555be0e2169e6e626b8eed281bf31d3b2fa28c99b11152034ed90472005b: Status 404 returned error can't find the container with id 3327555be0e2169e6e626b8eed281bf31d3b2fa28c99b11152034ed90472005b
Sep 12 00:16:40.819929 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.819881    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae99fad_f382_45ba_a89a_8cc391fb3d9f.slice/crio-5e621a34bcde1755af19db6e621f19865784338b1a50c0dc5e242ef309ed7cc8 WatchSource:0}: Error finding container 5e621a34bcde1755af19db6e621f19865784338b1a50c0dc5e242ef309ed7cc8: Status 404 returned error can't find the container with id 5e621a34bcde1755af19db6e621f19865784338b1a50c0dc5e242ef309ed7cc8
Sep 12 00:16:40.823885 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.823862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a021a5_f8de_48be_8456_1c71ef09a626.slice/crio-c6effd0820d92c0a9ef2ee492353c79517c6e2040e3d2c6829df2e67e187aaa4 WatchSource:0}: Error finding container c6effd0820d92c0a9ef2ee492353c79517c6e2040e3d2c6829df2e67e187aaa4: Status 404 returned error can't find the container with id c6effd0820d92c0a9ef2ee492353c79517c6e2040e3d2c6829df2e67e187aaa4
Sep 12 00:16:40.825575 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.825435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce54e557_520d_4218_b42b_2571f6e04a5b.slice/crio-292eaf327c1ceec35b5c033bb82f9452f1771ea8cdcaca3d00e8a4e1fc133709 WatchSource:0}: Error finding container 292eaf327c1ceec35b5c033bb82f9452f1771ea8cdcaca3d00e8a4e1fc133709: Status 404 returned error can't find the container with id 292eaf327c1ceec35b5c033bb82f9452f1771ea8cdcaca3d00e8a4e1fc133709
Sep 12 00:16:40.826326 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.826303    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936b2277_3619_42f3_aede_3c77c9e6285c.slice/crio-1a477eb9532fb6f805fd4afe895e107bcf9c227997ff2fafa75208883818884c WatchSource:0}: Error finding container 1a477eb9532fb6f805fd4afe895e107bcf9c227997ff2fafa75208883818884c: Status 404 returned error can't find the container with id 1a477eb9532fb6f805fd4afe895e107bcf9c227997ff2fafa75208883818884c
Sep 12 00:16:40.827402 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.827333    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592b7d6c_e39d_49c4_8d7b_4d4e15ae9b2d.slice/crio-34899b872a31b970965027416ba0172a9e4d19f420d6567f5508312c8b98a548 WatchSource:0}: Error finding container 34899b872a31b970965027416ba0172a9e4d19f420d6567f5508312c8b98a548: Status 404 returned error can't find the container with id 34899b872a31b970965027416ba0172a9e4d19f420d6567f5508312c8b98a548
Sep 12 00:16:40.828672 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.828211    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1cc4ee_f316_444d_af21_4fa2788a9a28.slice/crio-c2547c919a81ee68c3cabf7d5a6e0b0cbaf4649bd59edf093d8344317617ac51 WatchSource:0}: Error finding container c2547c919a81ee68c3cabf7d5a6e0b0cbaf4649bd59edf093d8344317617ac51: Status 404 returned error can't find the container with id c2547c919a81ee68c3cabf7d5a6e0b0cbaf4649bd59edf093d8344317617ac51
Sep 12 00:16:40.830760 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:16:40.830718    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b87268_b568_4b87_b832_4ff5a4d65a95.slice/crio-c48b64746c391947b9b897cb30a62535feb1eb1e598d8422489163616056bcc0 WatchSource:0}: Error finding container c48b64746c391947b9b897cb30a62535feb1eb1e598d8422489163616056bcc0: Status 404 returned error can't find the container with id c48b64746c391947b9b897cb30a62535feb1eb1e598d8422489163616056bcc0
Sep 12 00:16:41.040336 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.040139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:41.040531 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.040286    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:41.040531 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.040406    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:43.04039059 +0000 UTC m=+5.977697091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:41.140966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.140942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:41.141135 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.141092    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:41.141211 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.141147    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:43.14113062 +0000 UTC m=+6.078437128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:41.241694 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.241639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:41.241870 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.241796    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:16:41.241870 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.241824    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:16:41.241870 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.241837    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:41.242019 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.241896    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:43.241879518 +0000 UTC m=+6.179186019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:41.565756 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.565670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:41.566438 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.565816    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:41.568273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.568103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:41.568273 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:41.568220    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:41.583459 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.583384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sw4wh" event={"ID":"4ae99fad-f382-45ba-a89a-8cc391fb3d9f","Type":"ContainerStarted","Data":"5e621a34bcde1755af19db6e621f19865784338b1a50c0dc5e242ef309ed7cc8"}
Sep 12 00:16:41.595490 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.595458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gxj9b" event={"ID":"4353938a-ab7e-4fd7-b93c-2336a53ff5f9","Type":"ContainerStarted","Data":"9c65fab870d14059b6604ac20a78d2a965285b8f6a24d09bc58beaeb337325a5"}
Sep 12 00:16:41.602727 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.602697    2562 generic.go:356] "Generic (PLEG): container finished" podID="188d48a6c12b055e11435525c2a8acb7" containerID="7f36680bd59fef137528e31aba7beaf418f4ac3dfa9b3ad812fcfed5d5d0852c" exitCode=0
Sep 12 00:16:41.602862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.602783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal" event={"ID":"188d48a6c12b055e11435525c2a8acb7","Type":"ContainerDied","Data":"7f36680bd59fef137528e31aba7beaf418f4ac3dfa9b3ad812fcfed5d5d0852c"}
Sep 12 00:16:41.610213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.610181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rthv" event={"ID":"ce54e557-520d-4218-b42b-2571f6e04a5b","Type":"ContainerStarted","Data":"292eaf327c1ceec35b5c033bb82f9452f1771ea8cdcaca3d00e8a4e1fc133709"}
Sep 12 00:16:41.610344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.610295    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:11:40 +0000 UTC" deadline="2028-03-03 14:35:21.985013718 +0000 UTC"
Sep 12 00:16:41.610421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.610346    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12926h18m40.374671694s"
Sep 12 00:16:41.617050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.617023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerStarted","Data":"34899b872a31b970965027416ba0172a9e4d19f420d6567f5508312c8b98a548"}
Sep 12 00:16:41.620602 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.620576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nw9lf" event={"ID":"8a1cc4ee-f316-444d-af21-4fa2788a9a28","Type":"ContainerStarted","Data":"c2547c919a81ee68c3cabf7d5a6e0b0cbaf4649bd59edf093d8344317617ac51"}
Sep 12 00:16:41.626174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.626126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ttcd9" event={"ID":"a1a021a5-f8de-48be-8456-1c71ef09a626","Type":"ContainerStarted","Data":"c6effd0820d92c0a9ef2ee492353c79517c6e2040e3d2c6829df2e67e187aaa4"}
Sep 12 00:16:41.642019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.641713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77" event={"ID":"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd","Type":"ContainerStarted","Data":"3327555be0e2169e6e626b8eed281bf31d3b2fa28c99b11152034ed90472005b"}
Sep 12 00:16:41.652594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.652305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal" event={"ID":"7f8f2c475d91043fb3907657c68af0ed","Type":"ContainerStarted","Data":"1700e443c2dcf844e7219118f09fd3be723dbf226688d08d5a3702a7af6d6412"}
Sep 12 00:16:41.666850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.666792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"c48b64746c391947b9b897cb30a62535feb1eb1e598d8422489163616056bcc0"}
Sep 12 00:16:41.682343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:41.682258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7x4r8" event={"ID":"936b2277-3619-42f3-aede-3c77c9e6285c","Type":"ContainerStarted","Data":"1a477eb9532fb6f805fd4afe895e107bcf9c227997ff2fafa75208883818884c"}
Sep 12 00:16:42.566578 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:42.565667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:42.566578 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:42.565811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:42.709641 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:42.709564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal" event={"ID":"188d48a6c12b055e11435525c2a8acb7","Type":"ContainerStarted","Data":"4e91b4cf48eeb83b070c107695f1f5683016ba7904a7b7a4cfc90d4432b0102d"}
Sep 12 00:16:42.728656 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:42.727837    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-187.ec2.internal" podStartSLOduration=3.727812969 podStartE2EDuration="3.727812969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:42.72774678 +0000 UTC m=+5.665053305" watchObservedRunningTime="2026-09-12 00:16:42.727812969 +0000 UTC m=+5.665119495"
Sep 12 00:16:42.728656 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:42.728423    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-187.ec2.internal" podStartSLOduration=3.7284104449999997 podStartE2EDuration="3.728410445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:41.681748019 +0000 UTC m=+4.619054543" watchObservedRunningTime="2026-09-12 00:16:42.728410445 +0000 UTC m=+5.665716969"
Sep 12 00:16:43.057729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:43.057683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:43.057918 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.057902    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:43.058032 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.058013    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:47.057992299 +0000 UTC m=+9.995298815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:43.158567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:43.158512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:43.158768 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.158742    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:43.158844 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.158830    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:47.158808714 +0000 UTC m=+10.096115298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:43.260405 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:43.259716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:43.260405 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.259929    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:16:43.260405 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.259953    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:16:43.260405 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.259967    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:43.260405 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.260026    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:47.260007288 +0000 UTC m=+10.197313792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:43.567334 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:43.567298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:43.567858 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.567464    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:43.567922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:43.567866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:43.568024 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:43.567965    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:44.564743 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:44.564696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:44.564940 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:44.564879    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:45.564852 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:45.564817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:45.565348 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:45.564968    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:45.565348 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:45.565329    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:45.565488 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:45.565439    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:46.564463 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:46.564375    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:46.564662 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:46.564546    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:47.095205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:47.095167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:47.095846 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.095524    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:47.095846 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.095599    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:55.095581714 +0000 UTC m=+18.032888219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:47.196790 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:47.196744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:47.196975 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.196922    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:47.197039 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.197014    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:55.196991649 +0000 UTC m=+18.134298160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:47.297702 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:47.297656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:47.297906 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.297885    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:16:47.297987 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.297909    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:16:47.297987 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.297927    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:47.298096 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.298026    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:16:55.298004671 +0000 UTC m=+18.235311187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:47.565056 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:47.564977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:47.565217 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.565068    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:47.565545 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:47.565381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:47.565545 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:47.565501    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:48.563878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:48.563849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:48.564245 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:48.563976    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:49.565030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:49.564995    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:49.565521 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:49.565042    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:49.565521 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:49.565139    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:49.565521 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:49.565272    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:50.564653 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:50.564620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:50.564825 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:50.564740    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:51.564916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:51.564884    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:51.565281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:51.564941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:51.565281 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:51.565049    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:51.565498 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:51.565446    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:52.563776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:52.563741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:52.563960 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:52.563868    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:53.564199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:53.564112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:53.564676 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:53.564119    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:53.564676 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:53.564256    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:53.564676 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:53.564350    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:54.564204 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:54.564173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:54.564388 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:54.564301    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:55.158091 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:55.158053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:55.158240 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.158175    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:55.158240 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.158238    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:11.158220589 +0000 UTC m=+34.095527102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:16:55.258450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:55.258416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:55.258624 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.258594    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:55.258678 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.258671    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:11.258649308 +0000 UTC m=+34.195955829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:16:55.359741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:55.359665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:55.359892 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.359797    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:16:55.359892 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.359818    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:16:55.359892 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.359829    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:55.359892 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.359888    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:11.359869594 +0000 UTC m=+34.297176092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:16:55.563919 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:55.563887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:55.564077 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:55.563889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:55.564077 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.564029    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:55.564187 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:55.564118    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:56.564114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:56.564075    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:56.564554 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:56.564254    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:57.564812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:57.564723    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:57.565240 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:57.564860    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:57.565240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:57.564904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:57.565240 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:57.565022    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:58.564018 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:58.563989    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:16:58.564190 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:58.564103    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:16:59.564591 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.564279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:16:59.565232 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.564279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:16:59.565232 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:59.564669    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:16:59.565232 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:16:59.564723    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:16:59.749303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.749270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rthv" event={"ID":"ce54e557-520d-4218-b42b-2571f6e04a5b","Type":"ContainerStarted","Data":"d2c347a46fd4b9d3098079a0f4877424b50279faf45427451494d58ce347da92"}
Sep 12 00:16:59.750976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.750951    2562 generic.go:356] "Generic (PLEG): container finished" podID="592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d" containerID="34e8ff8b020825b2e92da5d432c93ae6af46ae3a8ce62c4b4fb9db659d4229da" exitCode=0
Sep 12 00:16:59.751086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.751031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerDied","Data":"34e8ff8b020825b2e92da5d432c93ae6af46ae3a8ce62c4b4fb9db659d4229da"}
Sep 12 00:16:59.752773 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.752666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nw9lf" event={"ID":"8a1cc4ee-f316-444d-af21-4fa2788a9a28","Type":"ContainerStarted","Data":"b74a54272b2c6c876d0cccd353b0e2c20017c321995d1f350129acb9d6e874a9"}
Sep 12 00:16:59.754203 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.754186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ttcd9" event={"ID":"a1a021a5-f8de-48be-8456-1c71ef09a626","Type":"ContainerStarted","Data":"be3e057ac17051c2994c35f875085592e8284b3349a345fbffd442e67e47e9e6"}
Sep 12 00:16:59.755792 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.755771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77" event={"ID":"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd","Type":"ContainerStarted","Data":"1cfae5eb50f170d2efebe57c83f13dab92325d24c6f30a6640b3e65ce91ddb23"}
Sep 12 00:16:59.759090 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.759060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"df8a90ebd99151a4389cfb9aafd95b51d504d7d0ed1b5e1158baa4bd6f2c5c67"}
Sep 12 00:16:59.759178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.759125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"0dcab8b2aead25b9936da3f4ce7f5b9ec0a827734dc7598657274d4b308335f0"}
Sep 12 00:16:59.759178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.759166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"8ddb2db2b3fafe08f7e5b03511486f72ff82f31e1c8921b2e6128ade35a97c61"}
Sep 12 00:16:59.759298 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.759196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"c3dc321b72802aaa36a6385b6359b09c124bc9f911ee22421ef4a3d41703d035"}
Sep 12 00:16:59.759298 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.759238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"dc9c16de8aabe73f7b1dbf14ba43777b1ade8ae4e746e65c98f423d436cb56ed"}
Sep 12 00:16:59.759506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.759463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"44a0e3d7f36220a3ebf63351ee5c0a1ab46eb692c2250ead6e136ea717d4b8e1"}
Sep 12 00:16:59.760628 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.760607    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sw4wh" event={"ID":"4ae99fad-f382-45ba-a89a-8cc391fb3d9f","Type":"ContainerStarted","Data":"78abfaf93c4a62447aa18de9ef5e74c2d2ead40754455b1163329df8559ab7ec"}
Sep 12 00:16:59.762090 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.762068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gxj9b" event={"ID":"4353938a-ab7e-4fd7-b93c-2336a53ff5f9","Type":"ContainerStarted","Data":"8d223ee99897d0705160d3d4c777842a7ddf6cb506bde80bc9d9b99d24a3f80d"}
Sep 12 00:16:59.765424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.765388    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2rthv" podStartSLOduration=11.114490898 podStartE2EDuration="21.765377445s" totalImagesPullingTime="10.650886547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:59.76487996 +0000 UTC m=+22.702186480" watchObservedRunningTime="2026-09-12 00:16:59.765377445 +0000 UTC m=+22.702683959"
Sep 12 00:16:59.780604 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.780547    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nw9lf" podStartSLOduration=3.903982144 podStartE2EDuration="21.780533544s" totalImagesPullingTime="17.8765514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:59.780031534 +0000 UTC m=+22.717338054" watchObservedRunningTime="2026-09-12 00:16:59.780533544 +0000 UTC m=+22.717840065"
Sep 12 00:16:59.804391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.804331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sw4wh" podStartSLOduration=3.852967597 podStartE2EDuration="21.804316697s" totalImagesPullingTime="17.9513491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:59.803797777 +0000 UTC m=+22.741104318" watchObservedRunningTime="2026-09-12 00:16:59.804316697 +0000 UTC m=+22.741623217"
Sep 12 00:16:59.819720 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.819674    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gxj9b" podStartSLOduration=3.932132668 podStartE2EDuration="21.819656417s" totalImagesPullingTime="17.887523749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:59.819544274 +0000 UTC m=+22.756850794" watchObservedRunningTime="2026-09-12 00:16:59.819656417 +0000 UTC m=+22.756962938"
Sep 12 00:16:59.858691 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.858637    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ttcd9" podStartSLOduration=3.978199283 podStartE2EDuration="21.858615593s" totalImagesPullingTime="17.88041631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:16:59.833977736 +0000 UTC m=+22.771284260" watchObservedRunningTime="2026-09-12 00:16:59.858615593 +0000 UTC m=+22.795922114"
Sep 12 00:16:59.951739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:16:59.951713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:00.012377 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.012290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:00.273881 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.273860    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 00:17:00.432858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.432823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:00.484767 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.484612    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T00:17:00.273877679Z","UUID":"24174b69-bbc2-4695-8775-4c69797fb51d","Handler":null,"Name":"","Endpoint":""}
Sep 12 00:17:00.485924 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.485904    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 00:17:00.486049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.485939    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 00:17:00.564105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.564031    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:00.564242 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:00.564134    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:17:00.766947 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.766901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7x4r8" event={"ID":"936b2277-3619-42f3-aede-3c77c9e6285c","Type":"ContainerStarted","Data":"5bdfd2848c6c2417f513f083bdf226ed0ba419fb7022b5c160b4821c86f42e57"}
Sep 12 00:17:00.769020 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.768989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77" event={"ID":"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd","Type":"ContainerStarted","Data":"4aecfd37638e73061707730dd19c392456b3079ec3d7e67661177ecd9c78408a"}
Sep 12 00:17:00.781559 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.781514    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7x4r8" podStartSLOduration=4.904293544 podStartE2EDuration="22.781499524s" totalImagesPullingTime="17.87720598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:00.781125193 +0000 UTC m=+23.718431714" watchObservedRunningTime="2026-09-12 00:17:00.781499524 +0000 UTC m=+23.718806048"
Sep 12 00:17:00.902793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.902689    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:00.978821 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:00.978787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:01.564175 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:01.563877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:01.564312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:01.563902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:01.564312 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:01.564231    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:17:01.564312 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:01.564272    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:17:01.772710 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:01.772672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77" event={"ID":"8a7191b5-a8fd-46a8-b20a-bdf7e3a84efd","Type":"ContainerStarted","Data":"0d7f2cc315ce61ed3e99b9c0df821ddf8daebeb7d7e763142de90b4c39b27c04"}
Sep 12 00:17:01.775890 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:01.775864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"f2c1481ced1e1e46a0659c4dc61bb38ca63e4e09e7a23cee47dd8babc8af3739"}
Sep 12 00:17:01.790189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:01.790151    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqw77" podStartSLOduration=3.750236176 podStartE2EDuration="23.790138541s" totalImagesPullingTime="20.039902365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:01.789949441 +0000 UTC m=+24.727255961" watchObservedRunningTime="2026-09-12 00:17:01.790138541 +0000 UTC m=+24.727445060"
Sep 12 00:17:02.564428 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:02.564394    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:02.564671 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:02.564527    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:17:02.792629 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:02.792597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:02.893383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:02.893214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:03.052675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:03.052645    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:17:03.053312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:03.053294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:17:03.564910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:03.564541    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:03.564910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:03.564587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:03.564910 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:03.564678    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:17:03.564910 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:03.564811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:17:03.784155 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:03.784081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" event={"ID":"e5b87268-b568-4b87-b832-4ff5a4d65a95","Type":"ContainerStarted","Data":"f792842a28076e72560124465ef739e084fd390cbd8b86509c8f8c5455b6e825"}
Sep 12 00:17:03.812104 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:03.812046    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4" podStartSLOduration=7.887793322 podStartE2EDuration="25.812029838s" totalImagesPullingTime="17.924236516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:03.810887693 +0000 UTC m=+26.748194212" watchObservedRunningTime="2026-09-12 00:17:03.812029838 +0000 UTC m=+26.749336343"
Sep 12 00:17:04.564010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:04.563973    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:04.564188 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:04.564090    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:17:04.788019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:04.787666    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:17:04.788019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:04.787758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:17:04.788019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:04.787816    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:17:04.819003 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:04.818510    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:17:04.825159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:04.824995    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:17:05.429057 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.428871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tm66"]
Sep 12 00:17:05.429211 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.429133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:05.429520 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:05.429240    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:17:05.431600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.431577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdzl6"]
Sep 12 00:17:05.431711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.431652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:05.431770 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:05.431724    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:17:05.443717 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.443665    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvk2b"]
Sep 12 00:17:05.443800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.443758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:05.443890 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:05.443870    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:17:05.601150 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.601117    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:05.654716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:05.654686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:06.614893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.614842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:06.670652 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.670614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:06.792232 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.792206    2562 generic.go:356] "Generic (PLEG): container finished" podID="592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d" containerID="dfdb732ab911d3d2adc9554239942ec780918be266994ac94aeaf42eb1f6bbd9" exitCode=0
Sep 12 00:17:06.792384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.792282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerDied","Data":"dfdb732ab911d3d2adc9554239942ec780918be266994ac94aeaf42eb1f6bbd9"}
Sep 12 00:17:06.984678 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.984604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:17:06.984835 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.984705    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 12 00:17:06.985186 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:06.985171    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2rthv"
Sep 12 00:17:07.045409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:07.045377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:07.566261 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:07.565759    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:07.566261 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:07.565865    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:17:07.566261 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:07.565978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:07.566261 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:07.566055    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:17:07.566261 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:07.566096    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:07.566261 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:07.566150    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:17:08.798375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:08.798183    2562 generic.go:356] "Generic (PLEG): container finished" podID="592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d" containerID="a99500749b15c5a1dcff4e585c36d62230542cf614a11ad78be566d10a89e963" exitCode=0
Sep 12 00:17:08.798770 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:08.798266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerDied","Data":"a99500749b15c5a1dcff4e585c36d62230542cf614a11ad78be566d10a89e963"}
Sep 12 00:17:09.564126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:09.564098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:09.564275 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:09.564098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:09.564275 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:09.564231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:09.564377 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:09.564207    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdzl6" podUID="471bf806-07c4-46b9-af50-16afee2dbbd3"
Sep 12 00:17:09.564377 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:09.564323    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tm66" podUID="be73927c-b191-4bef-8ded-c6bee22f4ae0"
Sep 12 00:17:09.564465 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:09.564439    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:17:09.901617 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:09.901527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:10.001254 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:10.001223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:10.607365 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:10.607341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:10.804842 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:10.804809    2562 generic.go:356] "Generic (PLEG): container finished" podID="592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d" containerID="78ca20c38c4b9550c84d4b1069a210fd296c823d3d506a5cba895594e12eaf07" exitCode=0
Sep 12 00:17:10.805013 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:10.804868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerDied","Data":"78ca20c38c4b9550c84d4b1069a210fd296c823d3d506a5cba895594e12eaf07"}
Sep 12 00:17:11.174211 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.174171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:11.174659 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.174286    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:17:11.174659 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.174346    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret podName:be73927c-b191-4bef-8ded-c6bee22f4ae0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:43.174330471 +0000 UTC m=+66.111636991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret") pod "global-pull-secret-syncer-7tm66" (UID: "be73927c-b191-4bef-8ded-c6bee22f4ae0") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:17:11.275396 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.275365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:11.275546 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.275523    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:17:11.275606 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.275593    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:43.275574179 +0000 UTC m=+66.212880696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:17:11.315877 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.315853    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-187.ec2.internal" event="NodeReady"
Sep 12 00:17:11.316004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.315992    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 00:17:11.358270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.358230    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f5hxx"]
Sep 12 00:17:11.361156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.361135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.361857 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.361837    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cpf2k"]
Sep 12 00:17:11.363891 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.363656    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 00:17:11.363891 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.363719    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 00:17:11.363891 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.363731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 00:17:11.364135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.363903    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ns4k9\""
Sep 12 00:17:11.364685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.364666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.366909 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.366870    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 00:17:11.367064 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.367047    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 00:17:11.367139 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.367082    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-knhwr\""
Sep 12 00:17:11.369910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.369886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5hxx"]
Sep 12 00:17:11.373388 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.373369    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cpf2k"]
Sep 12 00:17:11.376452 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.376427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:11.376637 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.376573    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:17:11.376637 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.376593    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:17:11.376637 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.376603    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rd4r9 for pod openshift-network-diagnostics/network-check-target-vdzl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:17:11.376637 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.376660    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9 podName:471bf806-07c4-46b9-af50-16afee2dbbd3 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:43.376642545 +0000 UTC m=+66.313949045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd4r9" (UniqueName: "kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9") pod "network-check-target-vdzl6" (UID: "471bf806-07c4-46b9-af50-16afee2dbbd3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:17:11.473992 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.473904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:11.477149 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.477119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt66r\" (UniqueName: \"kubernetes.io/projected/9e8227bd-18e8-4220-82fd-2e9878a7aecb-kube-api-access-qt66r\") pod \"ingress-canary-f5hxx\" (UID: \"9e8227bd-18e8-4220-82fd-2e9878a7aecb\") " pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.477282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.477156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.477282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.477184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba59c17-33b2-4642-a51c-e92070a474d0-tmp-dir\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.477282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.477224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qr8d\" (UniqueName: \"kubernetes.io/projected/9ba59c17-33b2-4642-a51c-e92070a474d0-kube-api-access-8qr8d\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.477438 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.477297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba59c17-33b2-4642-a51c-e92070a474d0-config-volume\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.477438 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.477345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e8227bd-18e8-4220-82fd-2e9878a7aecb-cert\") pod \"ingress-canary-f5hxx\" (UID: \"9e8227bd-18e8-4220-82fd-2e9878a7aecb\") " pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.564594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.564561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:11.564757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.564572    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:11.564757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.564570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:11.567260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.567200    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:17:11.567260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.567224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:17:11.567260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.567224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tdrkx\""
Sep 12 00:17:11.567505 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.567312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:17:11.567505 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.567336    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vv7j9\""
Sep 12 00:17:11.567625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.567605    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:17:11.577727 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.577707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba59c17-33b2-4642-a51c-e92070a474d0-config-volume\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.577846 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.577767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e8227bd-18e8-4220-82fd-2e9878a7aecb-cert\") pod \"ingress-canary-f5hxx\" (UID: \"9e8227bd-18e8-4220-82fd-2e9878a7aecb\") " pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.577911 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.577850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt66r\" (UniqueName: \"kubernetes.io/projected/9e8227bd-18e8-4220-82fd-2e9878a7aecb-kube-api-access-qt66r\") pod \"ingress-canary-f5hxx\" (UID: \"9e8227bd-18e8-4220-82fd-2e9878a7aecb\") " pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.577996 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.577980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.578039 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.578011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba59c17-33b2-4642-a51c-e92070a474d0-tmp-dir\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.578088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.578049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qr8d\" (UniqueName: \"kubernetes.io/projected/9ba59c17-33b2-4642-a51c-e92070a474d0-kube-api-access-8qr8d\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.578161 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.578130    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:11.578255 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:11.578200    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:12.07818462 +0000 UTC m=+35.015491139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:11.578345 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.578322    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba59c17-33b2-4642-a51c-e92070a474d0-tmp-dir\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.578419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.578399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba59c17-33b2-4642-a51c-e92070a474d0-config-volume\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.581817 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.581801    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e8227bd-18e8-4220-82fd-2e9878a7aecb-cert\") pod \"ingress-canary-f5hxx\" (UID: \"9e8227bd-18e8-4220-82fd-2e9878a7aecb\") " pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.586822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.586787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qr8d\" (UniqueName: \"kubernetes.io/projected/9ba59c17-33b2-4642-a51c-e92070a474d0-kube-api-access-8qr8d\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:11.586943 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.586819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt66r\" (UniqueName: \"kubernetes.io/projected/9e8227bd-18e8-4220-82fd-2e9878a7aecb-kube-api-access-qt66r\") pod \"ingress-canary-f5hxx\" (UID: \"9e8227bd-18e8-4220-82fd-2e9878a7aecb\") " pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.673938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.673902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5hxx"
Sep 12 00:17:11.955884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:11.955653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5hxx"]
Sep 12 00:17:11.960155 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:17:11.960125    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8227bd_18e8_4220_82fd_2e9878a7aecb.slice/crio-3aa5217f899df2e01ace3ef62ea3b8448319791a83d53a9554332db79f65e6cb WatchSource:0}: Error finding container 3aa5217f899df2e01ace3ef62ea3b8448319791a83d53a9554332db79f65e6cb: Status 404 returned error can't find the container with id 3aa5217f899df2e01ace3ef62ea3b8448319791a83d53a9554332db79f65e6cb
Sep 12 00:17:12.083655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:12.083565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:12.083828 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:12.083716    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:12.083893 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:12.083866    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:13.083845096 +0000 UTC m=+36.021151615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:12.541314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:12.541211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:12.599757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:12.599720    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:12.811879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:12.811783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5hxx" event={"ID":"9e8227bd-18e8-4220-82fd-2e9878a7aecb","Type":"ContainerStarted","Data":"3aa5217f899df2e01ace3ef62ea3b8448319791a83d53a9554332db79f65e6cb"}
Sep 12 00:17:13.091787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:13.091713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:13.091932 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:13.091867    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:13.091932 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:13.091930    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:15.09191559 +0000 UTC m=+38.029222088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:14.819075 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:14.818952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5hxx" event={"ID":"9e8227bd-18e8-4220-82fd-2e9878a7aecb","Type":"ContainerStarted","Data":"9d0fee68963443cbea5640f41a7363b0850385c74458b7fa6c499c6343b9efd0"}
Sep 12 00:17:14.834177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:14.834117    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f5hxx" podStartSLOduration=1.612595463 podStartE2EDuration="3.834095905s" totalImagesPullingTime="2.221500442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:14.833274669 +0000 UTC m=+37.770581204" watchObservedRunningTime="2026-09-12 00:17:14.834095905 +0000 UTC m=+37.771402425"
Sep 12 00:17:15.110608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:15.110520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:15.110751 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:15.110667    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:15.110751 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:15.110729    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:19.110713961 +0000 UTC m=+42.048020459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:17.815527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:17.813629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:17.828709 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:17.828677    2562 generic.go:356] "Generic (PLEG): container finished" podID="592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d" containerID="7feb98648191982ea4c198890790bd7feba37926407976e92fbb3dc50243dddb" exitCode=0
Sep 12 00:17:17.828851 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:17.828719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerDied","Data":"7feb98648191982ea4c198890790bd7feba37926407976e92fbb3dc50243dddb"}
Sep 12 00:17:17.874620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:17.874560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:18.833904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:18.833874    2562 generic.go:356] "Generic (PLEG): container finished" podID="592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d" containerID="d4f8c5305f165eb3f40893586686f2d99601465438a1ce062981724201b7ef46" exitCode=0
Sep 12 00:17:18.834281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:18.833921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerDied","Data":"d4f8c5305f165eb3f40893586686f2d99601465438a1ce062981724201b7ef46"}
Sep 12 00:17:19.136570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:19.136487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:19.136720 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:19.136582    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:19.136720 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:19.136649    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:27.136632723 +0000 UTC m=+50.073939221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:19.681585 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:19.681555    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:19.735670 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:19.735643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:19.840165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:19.840132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" event={"ID":"592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d","Type":"ContainerStarted","Data":"f8d51f24412c53afc600921989eeef07a0031053766cb77302ad64d41af8e689"}
Sep 12 00:17:19.861273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:19.861099    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mrzr4" podStartSLOduration=12.956445914 podStartE2EDuration="41.861083225s" totalImagesPullingTime="28.904637311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:19.859890308 +0000 UTC m=+42.797196828" watchObservedRunningTime="2026-09-12 00:17:19.861083225 +0000 UTC m=+42.798389745"
Sep 12 00:17:19.979928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:19.979843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:20.080124 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:20.080091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:20.131000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:20.130963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:24.519298 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:24.519269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:24.568881 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:24.568841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:27.185936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:27.185887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:27.186366 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:27.186038    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:27.186366 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:27.186103    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:17:43.186087203 +0000 UTC m=+66.123393701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:32.324448 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:32.324418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:36.803024 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:36.802989    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7wtr4"
Sep 12 00:17:36.963167 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:36.963134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:37.014682 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:37.014647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:37.727791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:37.727763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:37.775945 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:37.775920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:43.191093 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.191036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:17:43.191093 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.191092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:43.191564 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:43.191185    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:17:43.191564 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:43.191257    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:18:15.191238715 +0000 UTC m=+98.128545214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:17:43.193636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.193619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:17:43.205611 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.205582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be73927c-b191-4bef-8ded-c6bee22f4ae0-original-pull-secret\") pod \"global-pull-secret-syncer-7tm66\" (UID: \"be73927c-b191-4bef-8ded-c6bee22f4ae0\") " pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:43.291385 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.291339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:17:43.293925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.293909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:17:43.301895 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:43.301876    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:17:43.301954 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:17:43.301933    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:18:47.301916689 +0000 UTC m=+130.239223187 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : secret "metrics-daemon-secret" not found
Sep 12 00:17:43.380042 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.379998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tm66"
Sep 12 00:17:43.392071 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.392040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:43.394663 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.394643    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:17:43.404684 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.404662    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:17:43.415235 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.415212    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd4r9\" (UniqueName: \"kubernetes.io/projected/471bf806-07c4-46b9-af50-16afee2dbbd3-kube-api-access-rd4r9\") pod \"network-check-target-vdzl6\" (UID: \"471bf806-07c4-46b9-af50-16afee2dbbd3\") " pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:43.645179 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.645145    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tm66"]
Sep 12 00:17:43.648615 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:17:43.648591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe73927c_b191_4bef_8ded_c6bee22f4ae0.slice/crio-cfd59d8efba796867cc38d42e9194d7fd64389a5c76af80bf5822d3ef2d00192 WatchSource:0}: Error finding container cfd59d8efba796867cc38d42e9194d7fd64389a5c76af80bf5822d3ef2d00192: Status 404 returned error can't find the container with id cfd59d8efba796867cc38d42e9194d7fd64389a5c76af80bf5822d3ef2d00192
Sep 12 00:17:43.696043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.696015    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tdrkx\""
Sep 12 00:17:43.704014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.703997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:43.902729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.902648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7tm66" event={"ID":"be73927c-b191-4bef-8ded-c6bee22f4ae0","Type":"ContainerStarted","Data":"cfd59d8efba796867cc38d42e9194d7fd64389a5c76af80bf5822d3ef2d00192"}
Sep 12 00:17:43.971887 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:43.971854    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdzl6"]
Sep 12 00:17:43.975549 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:17:43.975525    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471bf806_07c4_46b9_af50_16afee2dbbd3.slice/crio-5e254dd50dc47877c9b576f58e3a0b3bf132f86464fefa23283879ecaaafcaa9 WatchSource:0}: Error finding container 5e254dd50dc47877c9b576f58e3a0b3bf132f86464fefa23283879ecaaafcaa9: Status 404 returned error can't find the container with id 5e254dd50dc47877c9b576f58e3a0b3bf132f86464fefa23283879ecaaafcaa9
Sep 12 00:17:44.639137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:44.639102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:44.701250 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:44.701216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:17:44.907287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:44.907193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdzl6" event={"ID":"471bf806-07c4-46b9-af50-16afee2dbbd3","Type":"ContainerStarted","Data":"5e254dd50dc47877c9b576f58e3a0b3bf132f86464fefa23283879ecaaafcaa9"}
Sep 12 00:17:48.920530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:48.920494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7tm66" event={"ID":"be73927c-b191-4bef-8ded-c6bee22f4ae0","Type":"ContainerStarted","Data":"ec3e37e47e0be6f688c5129fd1a66704b9b6a7e0453046c69d066479ef97b3ef"}
Sep 12 00:17:48.922135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:48.922112    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdzl6" event={"ID":"471bf806-07c4-46b9-af50-16afee2dbbd3","Type":"ContainerStarted","Data":"ddcaf15c65ad314aa22e3ca67a4b5312ade4b1bac4472c706c58844959af63df"}
Sep 12 00:17:48.922262 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:48.922237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:17:48.935365 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:48.935329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7tm66" podStartSLOduration=65.780590725 podStartE2EDuration="1m10.935315495s" totalImagesPullingTime="5.15472477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:48.935297357 +0000 UTC m=+71.872603879" watchObservedRunningTime="2026-09-12 00:17:48.935315495 +0000 UTC m=+71.872621993"
Sep 12 00:17:48.950234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:17:48.950194    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vdzl6" podStartSLOduration=66.127597216 podStartE2EDuration="1m10.950181907s" totalImagesPullingTime="4.822584691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:17:48.949664747 +0000 UTC m=+71.886971266" watchObservedRunningTime="2026-09-12 00:17:48.950181907 +0000 UTC m=+71.887488427"
Sep 12 00:18:08.747691 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.747578    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"]
Sep 12 00:18:08.751598 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.751578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:08.753993 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.753968    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 12 00:18:08.753993 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.753984    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 12 00:18:08.755007 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.754989    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-gp69g\""
Sep 12 00:18:08.755246 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.755229    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 12 00:18:08.755308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.755295    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 12 00:18:08.758028 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.758010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"]
Sep 12 00:18:08.864661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.864626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8ns\" (UniqueName: \"kubernetes.io/projected/6f3c229b-94a5-4bbc-bafc-79aeb367401d-kube-api-access-xt8ns\") pod \"managed-serviceaccount-addon-agent-666bb47cb-qphcm\" (UID: \"6f3c229b-94a5-4bbc-bafc-79aeb367401d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:08.864661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.864666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f3c229b-94a5-4bbc-bafc-79aeb367401d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-666bb47cb-qphcm\" (UID: \"6f3c229b-94a5-4bbc-bafc-79aeb367401d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:08.965214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.965182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8ns\" (UniqueName: \"kubernetes.io/projected/6f3c229b-94a5-4bbc-bafc-79aeb367401d-kube-api-access-xt8ns\") pod \"managed-serviceaccount-addon-agent-666bb47cb-qphcm\" (UID: \"6f3c229b-94a5-4bbc-bafc-79aeb367401d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:08.965214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.965217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f3c229b-94a5-4bbc-bafc-79aeb367401d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-666bb47cb-qphcm\" (UID: \"6f3c229b-94a5-4bbc-bafc-79aeb367401d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:08.967731 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.967713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f3c229b-94a5-4bbc-bafc-79aeb367401d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-666bb47cb-qphcm\" (UID: \"6f3c229b-94a5-4bbc-bafc-79aeb367401d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:08.973363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:08.973336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8ns\" (UniqueName: \"kubernetes.io/projected/6f3c229b-94a5-4bbc-bafc-79aeb367401d-kube-api-access-xt8ns\") pod \"managed-serviceaccount-addon-agent-666bb47cb-qphcm\" (UID: \"6f3c229b-94a5-4bbc-bafc-79aeb367401d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:09.071124 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:09.071093    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"
Sep 12 00:18:09.337555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:09.337463    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm"]
Sep 12 00:18:09.340251 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:18:09.340222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f3c229b_94a5_4bbc_bafc_79aeb367401d.slice/crio-fc6dd46cd5b66f70fe6c7c663119690016a3b0cb56e1ae027106abefe7821d00 WatchSource:0}: Error finding container fc6dd46cd5b66f70fe6c7c663119690016a3b0cb56e1ae027106abefe7821d00: Status 404 returned error can't find the container with id fc6dd46cd5b66f70fe6c7c663119690016a3b0cb56e1ae027106abefe7821d00
Sep 12 00:18:09.973593 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:09.973557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm" event={"ID":"6f3c229b-94a5-4bbc-bafc-79aeb367401d","Type":"ContainerStarted","Data":"fc6dd46cd5b66f70fe6c7c663119690016a3b0cb56e1ae027106abefe7821d00"}
Sep 12 00:18:11.980997 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:11.980969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm" event={"ID":"6f3c229b-94a5-4bbc-bafc-79aeb367401d","Type":"ContainerStarted","Data":"957d8e286aac0d4722d49ffa935182444f5396200834f5f58e6dd43ebef6edec"}
Sep 12 00:18:11.994966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:11.994539    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm" podStartSLOduration=1.450463638 podStartE2EDuration="3.994519008s" totalImagesPullingTime="2.54405537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:18:11.99398777 +0000 UTC m=+94.931294290" watchObservedRunningTime="2026-09-12 00:18:11.994519008 +0000 UTC m=+94.931825528"
Sep 12 00:18:15.207029 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:15.206984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:18:15.207429 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:18:15.207104    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:18:15.207429 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:18:15.207154    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:19:19.207141361 +0000 UTC m=+162.144447859 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:18:19.927432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:19.927401    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vdzl6"
Sep 12 00:18:47.320639 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:18:47.320590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:18:47.321134 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:18:47.320714    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:18:47.321134 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:18:47.320779    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs podName:5d1bde87-efc7-41eb-81a5-3da1c73fb6e6 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:49.320763562 +0000 UTC m=+252.258070060 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs") pod "network-metrics-daemon-qvk2b" (UID: "5d1bde87-efc7-41eb-81a5-3da1c73fb6e6") : secret "metrics-daemon-secret" not found
Sep 12 00:19:14.380242 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:19:14.380174    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cpf2k" podUID="9ba59c17-33b2-4642-a51c-e92070a474d0"
Sep 12 00:19:14.574731 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:19:14.574689    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qvk2b" podUID="5d1bde87-efc7-41eb-81a5-3da1c73fb6e6"
Sep 12 00:19:15.130076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:19:15.130048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:19:19.223100 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:19:19.223063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:19:19.223464 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:19:19.223184    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:19:19.223464 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:19:19.223244    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls podName:9ba59c17-33b2-4642-a51c-e92070a474d0 nodeName:}" failed. No retries permitted until 2026-09-12 00:21:21.2232302 +0000 UTC m=+284.160536703 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls") pod "dns-default-cpf2k" (UID: "9ba59c17-33b2-4642-a51c-e92070a474d0") : secret "dns-default-metrics-tls" not found
Sep 12 00:19:25.564176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:19:25.564135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:20:02.979578 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:02.979501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:12.263809 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:12.263718    2562 generic.go:356] "Generic (PLEG): container finished" podID="6f3c229b-94a5-4bbc-bafc-79aeb367401d" containerID="957d8e286aac0d4722d49ffa935182444f5396200834f5f58e6dd43ebef6edec" exitCode=255
Sep 12 00:20:12.263809 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:12.263760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm" event={"ID":"6f3c229b-94a5-4bbc-bafc-79aeb367401d","Type":"ContainerDied","Data":"957d8e286aac0d4722d49ffa935182444f5396200834f5f58e6dd43ebef6edec"}
Sep 12 00:20:12.269136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:12.269115    2562 scope.go:117] "RemoveContainer" containerID="957d8e286aac0d4722d49ffa935182444f5396200834f5f58e6dd43ebef6edec"
Sep 12 00:20:13.267422 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:13.267387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666bb47cb-qphcm" event={"ID":"6f3c229b-94a5-4bbc-bafc-79aeb367401d","Type":"ContainerStarted","Data":"6fe2341e5802abde6169e88d7556cd8d157f46c19c28c6117c28dbd666bfe01b"}
Sep 12 00:20:29.743262 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.743222    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"]
Sep 12 00:20:29.746036 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.746015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"
Sep 12 00:20:29.748902 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.748748    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kkgmm\""
Sep 12 00:20:29.749532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.749510    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"]
Sep 12 00:20:29.751952 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.751930    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66bc56dc8-bcvx4"]
Sep 12 00:20:29.752080 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.752060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:29.754559 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.754536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 12 00:20:29.754780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.754760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.754941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.754821    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 12 00:20:29.755211 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.755191    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7spgb\""
Sep 12 00:20:29.755854 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.755836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"]
Sep 12 00:20:29.757360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.757345    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 12 00:20:29.757659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.757646    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 12 00:20:29.757845 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.757830    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 12 00:20:29.757910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.757875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tp2x6\""
Sep 12 00:20:29.767027 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.767005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"]
Sep 12 00:20:29.767726 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.767703    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 12 00:20:29.772108 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.772090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66bc56dc8-bcvx4"]
Sep 12 00:20:29.789711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-image-registry-private-configuration\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.789711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-trusted-ca\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.789850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-bound-sa-token\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.789850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789759    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ktd\" (UniqueName: \"kubernetes.io/projected/6d1287ee-41d5-4638-bd4c-d28d66cb6e19-kube-api-access-44ktd\") pod \"network-check-source-6f8b768455-tmqft\" (UID: \"6d1287ee-41d5-4638-bd4c-d28d66cb6e19\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"
Sep 12 00:20:29.789850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6aba9d5b-aa98-4295-a190-707f83930beb-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:29.789850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-ca-trust-extracted\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.789850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:29.790002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv678\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-kube-api-access-cv678\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.790002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-certificates\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.790002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-installation-pull-secrets\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.790002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.789978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.842564 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.842529    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"]
Sep 12 00:20:29.845778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.845758    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"]
Sep 12 00:20:29.845970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.845899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:29.848534 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.848518    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-qd42h"]
Sep 12 00:20:29.848650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.848632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.848716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.848681    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 12 00:20:29.848716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.848693    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 12 00:20:29.848814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.848776    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8hf9j\""
Sep 12 00:20:29.849094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.849068    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:20:29.851007 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.850984    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:20:29.851154 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.851138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 12 00:20:29.851313 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.851294    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nv7jf\""
Sep 12 00:20:29.851405 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.851318    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-b5l7l"]
Sep 12 00:20:29.851487 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.851422    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 12 00:20:29.851487 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.851440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.851487 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.851443    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 12 00:20:29.854097 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.854080    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ssnhq\""
Sep 12 00:20:29.854200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.854183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.854554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.854536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 12 00:20:29.854678 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.854659    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 12 00:20:29.854777 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.854681    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 00:20:29.854777 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.854701    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 00:20:29.857632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.857612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 12 00:20:29.857740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.857717    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:20:29.857841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.857669    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 12 00:20:29.857970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.857948    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 12 00:20:29.858281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.858249    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wbsql\""
Sep 12 00:20:29.858950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.858929    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"]
Sep 12 00:20:29.860084 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.860064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"]
Sep 12 00:20:29.864030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.864005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qd42h"]
Sep 12 00:20:29.864526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.864501    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 12 00:20:29.864630 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.864523    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-b5l7l"]
Sep 12 00:20:29.865221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.865199    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 12 00:20:29.890442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4947b843-449b-4830-a0fb-66cbe4077a4e-config\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.890575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:29.890575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-image-registry-private-configuration\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.890575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-trusted-ca\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.890575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-bound-sa-token\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.890575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3562c34-1caf-42eb-9b2a-56abc7092a19-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqhmq\" (UniqueName: \"kubernetes.io/projected/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-kube-api-access-nqhmq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ktd\" (UniqueName: \"kubernetes.io/projected/6d1287ee-41d5-4638-bd4c-d28d66cb6e19-kube-api-access-44ktd\") pod \"network-check-source-6f8b768455-tmqft\" (UID: \"6d1287ee-41d5-4638-bd4c-d28d66cb6e19\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxw9d\" (UniqueName: \"kubernetes.io/projected/b3562c34-1caf-42eb-9b2a-56abc7092a19-kube-api-access-sxw9d\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3562c34-1caf-42eb-9b2a-56abc7092a19-service-ca-bundle\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv678\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-kube-api-access-cv678\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.890830 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.890821    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-certificates\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.890896    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert podName:6aba9d5b-aa98-4295-a190-707f83930beb nodeName:}" failed. No retries permitted until 2026-09-12 00:20:30.390875615 +0000 UTC m=+233.328182124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-xdfwv" (UID: "6aba9d5b-aa98-4295-a190-707f83930beb") : secret "networking-console-plugin-cert" not found
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-installation-pull-secrets\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.890987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.891112    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.891125    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bc56dc8-bcvx4: secret "image-registry-tls" not found
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3562c34-1caf-42eb-9b2a-56abc7092a19-serving-cert\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.891194 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.891172    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls podName:0c42d4f0-6198-4b4e-a453-e94a40cadfdf nodeName:}" failed. No retries permitted until 2026-09-12 00:20:30.391154578 +0000 UTC m=+233.328461096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls") pod "image-registry-66bc56dc8-bcvx4" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf") : secret "image-registry-tls" not found
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6aba9d5b-aa98-4295-a190-707f83930beb-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-ca-trust-extracted\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4947b843-449b-4830-a0fb-66cbe4077a4e-trusted-ca\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjz4\" (UniqueName: \"kubernetes.io/projected/4947b843-449b-4830-a0fb-66cbe4077a4e-kube-api-access-2pjz4\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4947b843-449b-4830-a0fb-66cbe4077a4e-serving-cert\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dct\" (UniqueName: \"kubernetes.io/projected/8db4b88e-22b4-4550-9014-b18afe769411-kube-api-access-n9dct\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-certificates\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3562c34-1caf-42eb-9b2a-56abc7092a19-tmp\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3562c34-1caf-42eb-9b2a-56abc7092a19-snapshots\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.891665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-ca-trust-extracted\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-trusted-ca\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.891995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.891854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6aba9d5b-aa98-4295-a190-707f83930beb-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:29.892899 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.892884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-image-registry-private-configuration\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.893324 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.893305    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-installation-pull-secrets\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.898921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.898899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-bound-sa-token\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.899991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.899974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv678\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-kube-api-access-cv678\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:29.900288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.900271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ktd\" (UniqueName: \"kubernetes.io/projected/6d1287ee-41d5-4638-bd4c-d28d66cb6e19-kube-api-access-44ktd\") pod \"network-check-source-6f8b768455-tmqft\" (UID: \"6d1287ee-41d5-4638-bd4c-d28d66cb6e19\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"
Sep 12 00:20:29.988076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.988044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:29.992120 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4947b843-449b-4830-a0fb-66cbe4077a4e-config\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.992188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:29.992188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3562c34-1caf-42eb-9b2a-56abc7092a19-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.992292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqhmq\" (UniqueName: \"kubernetes.io/projected/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-kube-api-access-nqhmq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.992292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxw9d\" (UniqueName: \"kubernetes.io/projected/b3562c34-1caf-42eb-9b2a-56abc7092a19-kube-api-access-sxw9d\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.992292 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.992261    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:20:29.992292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3562c34-1caf-42eb-9b2a-56abc7092a19-service-ca-bundle\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.992500 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:29.992326    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls podName:8db4b88e-22b4-4550-9014-b18afe769411 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:30.492307525 +0000 UTC m=+233.429614041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2dxns" (UID: "8db4b88e-22b4-4550-9014-b18afe769411") : secret "samples-operator-tls" not found
Sep 12 00:20:29.992500 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.992616 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3562c34-1caf-42eb-9b2a-56abc7092a19-serving-cert\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.992616 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4947b843-449b-4830-a0fb-66cbe4077a4e-trusted-ca\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.992715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjz4\" (UniqueName: \"kubernetes.io/projected/4947b843-449b-4830-a0fb-66cbe4077a4e-kube-api-access-2pjz4\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.992715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4947b843-449b-4830-a0fb-66cbe4077a4e-serving-cert\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.992715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dct\" (UniqueName: \"kubernetes.io/projected/8db4b88e-22b4-4550-9014-b18afe769411-kube-api-access-n9dct\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:29.992715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.992893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3562c34-1caf-42eb-9b2a-56abc7092a19-tmp\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.992893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3562c34-1caf-42eb-9b2a-56abc7092a19-snapshots\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.993001 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.992904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3562c34-1caf-42eb-9b2a-56abc7092a19-service-ca-bundle\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.993082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.993059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4947b843-449b-4830-a0fb-66cbe4077a4e-config\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.994003 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.993930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.994180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.994087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3562c34-1caf-42eb-9b2a-56abc7092a19-tmp\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.994336 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.994196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4947b843-449b-4830-a0fb-66cbe4077a4e-trusted-ca\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:29.998384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.994376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3562c34-1caf-42eb-9b2a-56abc7092a19-snapshots\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.998384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.994565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3562c34-1caf-42eb-9b2a-56abc7092a19-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.998384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.995263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3562c34-1caf-42eb-9b2a-56abc7092a19-serving-cert\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:29.998384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.995283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:29.999076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:29.999055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4947b843-449b-4830-a0fb-66cbe4077a4e-serving-cert\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:30.000896 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.000870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqhmq\" (UniqueName: \"kubernetes.io/projected/87f8b85f-b68a-47c2-abbd-091e63c3f3d1-kube-api-access-nqhmq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bjdzv\" (UID: \"87f8b85f-b68a-47c2-abbd-091e63c3f3d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:30.001283 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.001262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxw9d\" (UniqueName: \"kubernetes.io/projected/b3562c34-1caf-42eb-9b2a-56abc7092a19-kube-api-access-sxw9d\") pod \"insights-operator-5844b856db-qd42h\" (UID: \"b3562c34-1caf-42eb-9b2a-56abc7092a19\") " pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:30.001489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.001447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dct\" (UniqueName: \"kubernetes.io/projected/8db4b88e-22b4-4550-9014-b18afe769411-kube-api-access-n9dct\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:30.001848 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.001817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjz4\" (UniqueName: \"kubernetes.io/projected/4947b843-449b-4830-a0fb-66cbe4077a4e-kube-api-access-2pjz4\") pod \"console-operator-696884fb9c-b5l7l\" (UID: \"4947b843-449b-4830-a0fb-66cbe4077a4e\") " pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:30.054617 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.054593    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"
Sep 12 00:20:30.084700 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.084674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:30.165136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.165109    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"
Sep 12 00:20:30.172078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.172058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qd42h"
Sep 12 00:20:30.176881 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.176860    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:30.321483 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.321438    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-tmqft"]
Sep 12 00:20:30.324144 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:30.324119    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1287ee_41d5_4638_bd4c_d28d66cb6e19.slice/crio-2545abac972e570fc15c4a35e80c6188523a623e49ffa3c4ccf0aada63bc62d6 WatchSource:0}: Error finding container 2545abac972e570fc15c4a35e80c6188523a623e49ffa3c4ccf0aada63bc62d6: Status 404 returned error can't find the container with id 2545abac972e570fc15c4a35e80c6188523a623e49ffa3c4ccf0aada63bc62d6
Sep 12 00:20:30.396266 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.396235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:30.396388 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.396361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:30.396388 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.396376    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:20:30.396466 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.396396    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bc56dc8-bcvx4: secret "image-registry-tls" not found
Sep 12 00:20:30.396466 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.396439    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls podName:0c42d4f0-6198-4b4e-a453-e94a40cadfdf nodeName:}" failed. No retries permitted until 2026-09-12 00:20:31.39642379 +0000 UTC m=+234.333730303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls") pod "image-registry-66bc56dc8-bcvx4" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf") : secret "image-registry-tls" not found
Sep 12 00:20:30.396563 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.396511    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:20:30.396605 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.396567    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert podName:6aba9d5b-aa98-4295-a190-707f83930beb nodeName:}" failed. No retries permitted until 2026-09-12 00:20:31.396550996 +0000 UTC m=+234.333857494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-xdfwv" (UID: "6aba9d5b-aa98-4295-a190-707f83930beb") : secret "networking-console-plugin-cert" not found
Sep 12 00:20:30.436911 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.436886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv"]
Sep 12 00:20:30.439924 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:30.439900    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f8b85f_b68a_47c2_abbd_091e63c3f3d1.slice/crio-230ccef668a245e2f2c401734548e455298e1c70392bc0e8f163598bf68f59eb WatchSource:0}: Error finding container 230ccef668a245e2f2c401734548e455298e1c70392bc0e8f163598bf68f59eb: Status 404 returned error can't find the container with id 230ccef668a245e2f2c401734548e455298e1c70392bc0e8f163598bf68f59eb
Sep 12 00:20:30.460609 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.460588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-b5l7l"]
Sep 12 00:20:30.463288 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:30.463261    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4947b843_449b_4830_a0fb_66cbe4077a4e.slice/crio-0ab475ae9d1257e57ed8bb9fff80c47be51d84c2159c1670002aff89b1d15709 WatchSource:0}: Error finding container 0ab475ae9d1257e57ed8bb9fff80c47be51d84c2159c1670002aff89b1d15709: Status 404 returned error can't find the container with id 0ab475ae9d1257e57ed8bb9fff80c47be51d84c2159c1670002aff89b1d15709
Sep 12 00:20:30.464178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.464088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qd42h"]
Sep 12 00:20:30.467027 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:30.467005    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3562c34_1caf_42eb_9b2a_56abc7092a19.slice/crio-fca20b09a6d474f280eed4fa3fdb47f0f5b95f7c05d76117d65673862512724c WatchSource:0}: Error finding container fca20b09a6d474f280eed4fa3fdb47f0f5b95f7c05d76117d65673862512724c: Status 404 returned error can't find the container with id fca20b09a6d474f280eed4fa3fdb47f0f5b95f7c05d76117d65673862512724c
Sep 12 00:20:30.497428 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:30.497368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:30.497558 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.497541    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:20:30.497632 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:30.497609    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls podName:8db4b88e-22b4-4550-9014-b18afe769411 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:31.497588169 +0000 UTC m=+234.434894668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2dxns" (UID: "8db4b88e-22b4-4550-9014-b18afe769411") : secret "samples-operator-tls" not found
Sep 12 00:20:31.315810 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.315716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft" event={"ID":"6d1287ee-41d5-4638-bd4c-d28d66cb6e19","Type":"ContainerStarted","Data":"163248968b80ea2d9e3df116dfe0b73668e87a3fb134c5443cb70ce89fde4a0f"}
Sep 12 00:20:31.315810 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.315773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft" event={"ID":"6d1287ee-41d5-4638-bd4c-d28d66cb6e19","Type":"ContainerStarted","Data":"2545abac972e570fc15c4a35e80c6188523a623e49ffa3c4ccf0aada63bc62d6"}
Sep 12 00:20:31.317650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.317580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" event={"ID":"4947b843-449b-4830-a0fb-66cbe4077a4e","Type":"ContainerStarted","Data":"0ab475ae9d1257e57ed8bb9fff80c47be51d84c2159c1670002aff89b1d15709"}
Sep 12 00:20:31.320255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.320010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qd42h" event={"ID":"b3562c34-1caf-42eb-9b2a-56abc7092a19","Type":"ContainerStarted","Data":"fca20b09a6d474f280eed4fa3fdb47f0f5b95f7c05d76117d65673862512724c"}
Sep 12 00:20:31.322628 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.322538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv" event={"ID":"87f8b85f-b68a-47c2-abbd-091e63c3f3d1","Type":"ContainerStarted","Data":"230ccef668a245e2f2c401734548e455298e1c70392bc0e8f163598bf68f59eb"}
Sep 12 00:20:31.332159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.332115    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-tmqft" podStartSLOduration=2.332096922 podStartE2EDuration="2.332096922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:31.3306027 +0000 UTC m=+234.267909221" watchObservedRunningTime="2026-09-12 00:20:31.332096922 +0000 UTC m=+234.269403444"
Sep 12 00:20:31.406014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.405930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:31.406014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.406003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:31.406258 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.406139    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:20:31.406258 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.406175    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:20:31.406258 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.406191    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bc56dc8-bcvx4: secret "image-registry-tls" not found
Sep 12 00:20:31.406258 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.406216    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert podName:6aba9d5b-aa98-4295-a190-707f83930beb nodeName:}" failed. No retries permitted until 2026-09-12 00:20:33.40619361 +0000 UTC m=+236.343500113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-xdfwv" (UID: "6aba9d5b-aa98-4295-a190-707f83930beb") : secret "networking-console-plugin-cert" not found
Sep 12 00:20:31.406258 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.406237    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls podName:0c42d4f0-6198-4b4e-a453-e94a40cadfdf nodeName:}" failed. No retries permitted until 2026-09-12 00:20:33.406222619 +0000 UTC m=+236.343529125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls") pod "image-registry-66bc56dc8-bcvx4" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf") : secret "image-registry-tls" not found
Sep 12 00:20:31.507402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.507362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:31.507585 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.507565    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:20:31.507652 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:31.507634    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls podName:8db4b88e-22b4-4550-9014-b18afe769411 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:33.50761664 +0000 UTC m=+236.444923139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2dxns" (UID: "8db4b88e-22b4-4550-9014-b18afe769411") : secret "samples-operator-tls" not found
Sep 12 00:20:31.993015 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:31.992949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:32.043682 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:32.043648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:33.383788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.383753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:33.424668 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.424632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:33.424847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.424760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:33.424847 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.424802    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:20:33.424847 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.424822    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bc56dc8-bcvx4: secret "image-registry-tls" not found
Sep 12 00:20:33.425014 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.424888    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:20:33.425014 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.424891    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls podName:0c42d4f0-6198-4b4e-a453-e94a40cadfdf nodeName:}" failed. No retries permitted until 2026-09-12 00:20:37.42487079 +0000 UTC m=+240.362177300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls") pod "image-registry-66bc56dc8-bcvx4" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf") : secret "image-registry-tls" not found
Sep 12 00:20:33.425014 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.424953    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert podName:6aba9d5b-aa98-4295-a190-707f83930beb nodeName:}" failed. No retries permitted until 2026-09-12 00:20:37.424936349 +0000 UTC m=+240.362242857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-xdfwv" (UID: "6aba9d5b-aa98-4295-a190-707f83930beb") : secret "networking-console-plugin-cert" not found
Sep 12 00:20:33.485122 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.485092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:33.525996 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.525965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:33.526123 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.526108    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:20:33.526174 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:33.526165    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls podName:8db4b88e-22b4-4550-9014-b18afe769411 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:37.52615094 +0000 UTC m=+240.463457438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2dxns" (UID: "8db4b88e-22b4-4550-9014-b18afe769411") : secret "samples-operator-tls" not found
Sep 12 00:20:33.819622 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.819589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:33.922093 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:33.922015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:34.097904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.097870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:34.196309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.196232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:34.342047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.342018    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/0.log"
Sep 12 00:20:34.342230 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.342057    2562 generic.go:356] "Generic (PLEG): container finished" podID="4947b843-449b-4830-a0fb-66cbe4077a4e" containerID="89a4cb80c025d5dc834b233d685a094bc42c309b373d70b80e15c943ba8b52f9" exitCode=255
Sep 12 00:20:34.342230 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.342120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" event={"ID":"4947b843-449b-4830-a0fb-66cbe4077a4e","Type":"ContainerDied","Data":"89a4cb80c025d5dc834b233d685a094bc42c309b373d70b80e15c943ba8b52f9"}
Sep 12 00:20:34.342426 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.342407    2562 scope.go:117] "RemoveContainer" containerID="89a4cb80c025d5dc834b233d685a094bc42c309b373d70b80e15c943ba8b52f9"
Sep 12 00:20:34.343821 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.343794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qd42h" event={"ID":"b3562c34-1caf-42eb-9b2a-56abc7092a19","Type":"ContainerStarted","Data":"787cf0d67d8c395bcdca0ee02f61205c3c2dc8180595a92a0640253e7997016d"}
Sep 12 00:20:34.345575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.345548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv" event={"ID":"87f8b85f-b68a-47c2-abbd-091e63c3f3d1","Type":"ContainerStarted","Data":"ecfb3ba0248bf0c7c3287ce4f6dd9b07952b54e6ada0d371dbb48afb798defc3"}
Sep 12 00:20:34.372920 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.372846    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv" podStartSLOduration=2.275548244 podStartE2EDuration="5.372826242s" totalImagesPullingTime="3.097277998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:34.371555536 +0000 UTC m=+237.308862055" watchObservedRunningTime="2026-09-12 00:20:34.372826242 +0000 UTC m=+237.310132763"
Sep 12 00:20:34.387153 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:34.386794    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-qd42h" podStartSLOduration=2.313564944 podStartE2EDuration="5.386775942s" totalImagesPullingTime="3.073210998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:34.386352492 +0000 UTC m=+237.323659015" watchObservedRunningTime="2026-09-12 00:20:34.386775942 +0000 UTC m=+237.324082462"
Sep 12 00:20:35.009704 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.009670    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l64ss"]
Sep 12 00:20:35.012708 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.012689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.015138 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.015116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 00:20:35.015228 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.015137    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pwzwl\""
Sep 12 00:20:35.015228 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.015165    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 00:20:35.016217 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.016195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:35.022496 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.022459    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l64ss"]
Sep 12 00:20:35.037297 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.037268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvm4j\" (UniqueName: \"kubernetes.io/projected/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-api-access-pvm4j\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.037417 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.037328    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6ae1947-3afa-47ca-913f-9744a21324f0-crio-socket\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.037495 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.037435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.037571 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.037554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6ae1947-3afa-47ca-913f-9744a21324f0-data-volume\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.037610 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.037587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.071642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.071614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:35.138778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.138744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvm4j\" (UniqueName: \"kubernetes.io/projected/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-api-access-pvm4j\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.138968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.138790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6ae1947-3afa-47ca-913f-9744a21324f0-crio-socket\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.138968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.138825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.138968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.138866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6ae1947-3afa-47ca-913f-9744a21324f0-data-volume\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.138968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.138883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.138968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.138907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6ae1947-3afa-47ca-913f-9744a21324f0-crio-socket\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.139181 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.138986    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:35.139181 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.139035    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:20:35.139181 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.139046    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:35.639032309 +0000 UTC m=+238.576338807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:35.139181 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.139098    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:35.63908159 +0000 UTC m=+238.576388299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : configmap "kube-rbac-proxy" not found
Sep 12 00:20:35.139310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.139245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6ae1947-3afa-47ca-913f-9744a21324f0-data-volume\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.147127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.147105    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvm4j\" (UniqueName: \"kubernetes.io/projected/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-api-access-pvm4j\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.349623 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.349599    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:20:35.350032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.350013    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/0.log"
Sep 12 00:20:35.350130 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.350054    2562 generic.go:356] "Generic (PLEG): container finished" podID="4947b843-449b-4830-a0fb-66cbe4077a4e" containerID="9e67ec53a2389faf508919bc65ba41db934c4b0f7d2428fa1f02477b13381edd" exitCode=255
Sep 12 00:20:35.350183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.350136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" event={"ID":"4947b843-449b-4830-a0fb-66cbe4077a4e","Type":"ContainerDied","Data":"9e67ec53a2389faf508919bc65ba41db934c4b0f7d2428fa1f02477b13381edd"}
Sep 12 00:20:35.350253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.350189    2562 scope.go:117] "RemoveContainer" containerID="89a4cb80c025d5dc834b233d685a094bc42c309b373d70b80e15c943ba8b52f9"
Sep 12 00:20:35.350447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.350430    2562 scope.go:117] "RemoveContainer" containerID="9e67ec53a2389faf508919bc65ba41db934c4b0f7d2428fa1f02477b13381edd"
Sep 12 00:20:35.350647 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.350626    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-b5l7l_openshift-console-operator(4947b843-449b-4830-a0fb-66cbe4077a4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" podUID="4947b843-449b-4830-a0fb-66cbe4077a4e"
Sep 12 00:20:35.447031 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.447004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:35.473710 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.473679    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"]
Sep 12 00:20:35.477883 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.477861    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"
Sep 12 00:20:35.480188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.480168    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 12 00:20:35.480296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.480202    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-x9rvs\""
Sep 12 00:20:35.480296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.480238    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 12 00:20:35.486502 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.486461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"]
Sep 12 00:20:35.542900 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.542872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d944\" (UniqueName: \"kubernetes.io/projected/0e565eed-f414-408a-b225-3a1c948d9826-kube-api-access-8d944\") pod \"migrator-8b4d697c5-xplln\" (UID: \"0e565eed-f414-408a-b225-3a1c948d9826\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"
Sep 12 00:20:35.546958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.546941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:35.643526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.643434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.643526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.643465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d944\" (UniqueName: \"kubernetes.io/projected/0e565eed-f414-408a-b225-3a1c948d9826-kube-api-access-8d944\") pod \"migrator-8b4d697c5-xplln\" (UID: \"0e565eed-f414-408a-b225-3a1c948d9826\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"
Sep 12 00:20:35.643526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.643518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:35.643736 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.643568    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:35.643736 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.643620    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:36.643606746 +0000 UTC m=+239.580913244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:35.643736 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.643645    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:20:35.643736 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:35.643695    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:36.643680352 +0000 UTC m=+239.580986857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : configmap "kube-rbac-proxy" not found
Sep 12 00:20:35.647539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.647520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:35.651324 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.651295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d944\" (UniqueName: \"kubernetes.io/projected/0e565eed-f414-408a-b225-3a1c948d9826-kube-api-access-8d944\") pod \"migrator-8b4d697c5-xplln\" (UID: \"0e565eed-f414-408a-b225-3a1c948d9826\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"
Sep 12 00:20:35.746755 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.746724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:35.787637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:35.787613    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"
Sep 12 00:20:36.056152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.056128    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln"]
Sep 12 00:20:36.058571 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:36.058538    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e565eed_f414_408a_b225_3a1c948d9826.slice/crio-2e6cb32bbcc30740ec398165145b824501b553d78b248dffecbae2e97fce82f8 WatchSource:0}: Error finding container 2e6cb32bbcc30740ec398165145b824501b553d78b248dffecbae2e97fce82f8: Status 404 returned error can't find the container with id 2e6cb32bbcc30740ec398165145b824501b553d78b248dffecbae2e97fce82f8
Sep 12 00:20:36.355204 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.355178    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:20:36.355536 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.355519    2562 scope.go:117] "RemoveContainer" containerID="9e67ec53a2389faf508919bc65ba41db934c4b0f7d2428fa1f02477b13381edd"
Sep 12 00:20:36.355695 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:36.355673    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-b5l7l_openshift-console-operator(4947b843-449b-4830-a0fb-66cbe4077a4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" podUID="4947b843-449b-4830-a0fb-66cbe4077a4e"
Sep 12 00:20:36.356483 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.356447    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln" event={"ID":"0e565eed-f414-408a-b225-3a1c948d9826","Type":"ContainerStarted","Data":"2e6cb32bbcc30740ec398165145b824501b553d78b248dffecbae2e97fce82f8"}
Sep 12 00:20:36.398977 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.398943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:36.651308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.651228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:36.651308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:36.651283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:36.651730 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:36.651341    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:20:36.651730 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:36.651391    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:38.651377252 +0000 UTC m=+241.588683750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : configmap "kube-rbac-proxy" not found
Sep 12 00:20:36.651730 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:36.651389    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:36.651730 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:36.651458    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:38.651437484 +0000 UTC m=+241.588743995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:37.178457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:37.178431    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ttcd9_a1a021a5-f8de-48be-8456-1c71ef09a626/dns-node-resolver/0.log"
Sep 12 00:20:37.458029 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:37.458001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:37.458144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:37.458128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:37.458213 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.458157    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:20:37.458213 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.458176    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bc56dc8-bcvx4: secret "image-registry-tls" not found
Sep 12 00:20:37.458316 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.458228    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls podName:0c42d4f0-6198-4b4e-a453-e94a40cadfdf nodeName:}" failed. No retries permitted until 2026-09-12 00:20:45.458214922 +0000 UTC m=+248.395521424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls") pod "image-registry-66bc56dc8-bcvx4" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf") : secret "image-registry-tls" not found
Sep 12 00:20:37.458316 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.458259    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:20:37.458398 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.458316    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert podName:6aba9d5b-aa98-4295-a190-707f83930beb nodeName:}" failed. No retries permitted until 2026-09-12 00:20:45.45830219 +0000 UTC m=+248.395608693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-xdfwv" (UID: "6aba9d5b-aa98-4295-a190-707f83930beb") : secret "networking-console-plugin-cert" not found
Sep 12 00:20:37.559094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:37.559061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:37.559239 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.559196    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 12 00:20:37.559290 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:37.559261    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls podName:8db4b88e-22b4-4550-9014-b18afe769411 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:45.559246926 +0000 UTC m=+248.496553449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-2dxns" (UID: "8db4b88e-22b4-4550-9014-b18afe769411") : secret "samples-operator-tls" not found
Sep 12 00:20:37.778892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:37.778823    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gxj9b_4353938a-ab7e-4fd7-b93c-2336a53ff5f9/node-ca/0.log"
Sep 12 00:20:37.946631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:37.946597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:38.097318 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.097287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:38.223745 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.223718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:38.325561 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.325528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:38.369387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.369313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln" event={"ID":"0e565eed-f414-408a-b225-3a1c948d9826","Type":"ContainerStarted","Data":"6b4a9f2cf7d85310a78f6868414d4b450ce9814e73947c212d03c869455228aa"}
Sep 12 00:20:38.369387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.369360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln" event={"ID":"0e565eed-f414-408a-b225-3a1c948d9826","Type":"ContainerStarted","Data":"92ea0b42c8e567642d32b209cf57343b378037e45ee15ea2399f3c2b22ba46b6"}
Sep 12 00:20:38.386088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.386046    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xplln" podStartSLOduration=2.080709272 podStartE2EDuration="3.386032271s" totalImagesPullingTime="1.305322999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:38.385419942 +0000 UTC m=+241.322726463" watchObservedRunningTime="2026-09-12 00:20:38.386032271 +0000 UTC m=+241.323338790"
Sep 12 00:20:38.395595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.395572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:38.451086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.451060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:38.582159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.582135    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5hxx_9e8227bd-18e8-4220-82fd-2e9878a7aecb/serve-healthcheck-canary/0.log"
Sep 12 00:20:38.668183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.668110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:38.668183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:38.668170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:38.668345 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:38.668263    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:38.668345 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:38.668317    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:42.668304695 +0000 UTC m=+245.605611207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:38.668345 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:38.668266    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:20:38.668451 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:38.668382    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:42.668370327 +0000 UTC m=+245.605676840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : configmap "kube-rbac-proxy" not found
Sep 12 00:20:40.177199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.177164    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:40.177594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.177220    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:40.177594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.177558    2562 scope.go:117] "RemoveContainer" containerID="9e67ec53a2389faf508919bc65ba41db934c4b0f7d2428fa1f02477b13381edd"
Sep 12 00:20:40.177714 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:40.177697    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-b5l7l_openshift-console-operator(4947b843-449b-4830-a0fb-66cbe4077a4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" podUID="4947b843-449b-4830-a0fb-66cbe4077a4e"
Sep 12 00:20:40.301434 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.301403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:40.365413 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.365378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:40.549191 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.549114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:40.651465 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.651436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:40.852091 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:40.852060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:41.246537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:41.246443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:41.818836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:41.818806    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:41.869908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:41.869880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:42.450012 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:42.449984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:42.684683 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:42.684653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:42.702253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:42.702193    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:42.702384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:42.702273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:42.702384 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:42.702320    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:20:42.702384 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:42.702374    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:42.702566 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:42.702387    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:50.702369049 +0000 UTC m=+253.639675554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : configmap "kube-rbac-proxy" not found
Sep 12 00:20:42.702566 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:42.702411    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls podName:d6ae1947-3afa-47ca-913f-9744a21324f0 nodeName:}" failed. No retries permitted until 2026-09-12 00:20:50.702400806 +0000 UTC m=+253.639707303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l64ss" (UID: "d6ae1947-3afa-47ca-913f-9744a21324f0") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:20:42.851836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:42.851808    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:43.050419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:43.050336    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:43.252454 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:43.252422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:44.052334 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:44.052301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:44.244059 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:44.244031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:44.468674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:44.468643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:44.644003 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:44.643974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:44.850309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:44.850282    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:45.254308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.254215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:45.524813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.524734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:20:45.524813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.524787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:45.525002 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:45.524877    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 12 00:20:45.525002 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:20:45.524939    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert podName:6aba9d5b-aa98-4295-a190-707f83930beb nodeName:}" failed. No retries permitted until 2026-09-12 00:21:01.524923584 +0000 UTC m=+264.462230083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-xdfwv" (UID: "6aba9d5b-aa98-4295-a190-707f83930beb") : secret "networking-console-plugin-cert" not found
Sep 12 00:20:45.527089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.527067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"image-registry-66bc56dc8-bcvx4\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") " pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:45.625922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.625893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:45.628073 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.628050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db4b88e-22b4-4550-9014-b18afe769411-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-2dxns\" (UID: \"8db4b88e-22b4-4550-9014-b18afe769411\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:45.674080 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.674055    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tp2x6\""
Sep 12 00:20:45.682071 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.682048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:45.759342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.759321    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8hf9j\""
Sep 12 00:20:45.767181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.767160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"
Sep 12 00:20:45.841559 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.841522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:45.950516 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:45.950494    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66bc56dc8-bcvx4"]
Sep 12 00:20:45.952704 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:45.952667    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c42d4f0_6198_4b4e_a453_e94a40cadfdf.slice/crio-25e38f76222b3fd63b92d29750320bfddb6d4737eabab2b544f263f1428a8603 WatchSource:0}: Error finding container 25e38f76222b3fd63b92d29750320bfddb6d4737eabab2b544f263f1428a8603: Status 404 returned error can't find the container with id 25e38f76222b3fd63b92d29750320bfddb6d4737eabab2b544f263f1428a8603
Sep 12 00:20:46.034424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.034357    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns"]
Sep 12 00:20:46.241420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.241386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:46.394189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.394152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" event={"ID":"0c42d4f0-6198-4b4e-a453-e94a40cadfdf","Type":"ContainerStarted","Data":"bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a"}
Sep 12 00:20:46.394606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.394199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" event={"ID":"0c42d4f0-6198-4b4e-a453-e94a40cadfdf","Type":"ContainerStarted","Data":"25e38f76222b3fd63b92d29750320bfddb6d4737eabab2b544f263f1428a8603"}
Sep 12 00:20:46.394606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.394246    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:20:46.395373 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.395352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns" event={"ID":"8db4b88e-22b4-4550-9014-b18afe769411","Type":"ContainerStarted","Data":"5a9149de22cc6b94ed27f18e72b25e52b3c106f947d7663c86d4f3a921cc3537"}
Sep 12 00:20:46.413188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.413133    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" podStartSLOduration=17.413117276 podStartE2EDuration="17.413117276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:46.412786494 +0000 UTC m=+249.350093014" watchObservedRunningTime="2026-09-12 00:20:46.413117276 +0000 UTC m=+249.350423798"
Sep 12 00:20:46.443115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.443090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:46.846141 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:46.846104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:47.016864 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:47.016824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:47.443216 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:47.443161    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:47.846151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:47.846115    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:48.052123 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.052049    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:48.136916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.136884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:48.217766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.217740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:48.403861 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.403831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns" event={"ID":"8db4b88e-22b4-4550-9014-b18afe769411","Type":"ContainerStarted","Data":"2364ad986b42896e0057ef7ad41f2c7fc189146cadfc21b39b725134c1ba8763"}
Sep 12 00:20:48.404009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.403880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns" event={"ID":"8db4b88e-22b4-4550-9014-b18afe769411","Type":"ContainerStarted","Data":"dbe351287db6cd2aa0e1e2cba2f4d987c8f1643fa65df5217842a51432e06516"}
Sep 12 00:20:48.421814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.421771    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-2dxns" podStartSLOduration=17.811997303 podStartE2EDuration="19.42175642s" totalImagesPullingTime="1.609759117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:48.420291192 +0000 UTC m=+251.357597712" watchObservedRunningTime="2026-09-12 00:20:48.42175642 +0000 UTC m=+251.359062988"
Sep 12 00:20:48.439957 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.439927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:48.643375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:48.643349    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:49.050414 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:49.050356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:49.356376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:49.356340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:20:49.358576 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:49.358553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d1bde87-efc7-41eb-81a5-3da1c73fb6e6-metrics-certs\") pod \"network-metrics-daemon-qvk2b\" (UID: \"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6\") " pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:20:49.567633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:49.567604    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vv7j9\""
Sep 12 00:20:49.575381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:49.575363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvk2b"
Sep 12 00:20:49.839946 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:49.839922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvk2b"]
Sep 12 00:20:49.842231 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:49.842203    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1bde87_efc7_41eb_81a5_3da1c73fb6e6.slice/crio-d42453e634d5e3923ad752d3b69f4f488f5f29667826b27aea48afba3a6d18c4 WatchSource:0}: Error finding container d42453e634d5e3923ad752d3b69f4f488f5f29667826b27aea48afba3a6d18c4: Status 404 returned error can't find the container with id d42453e634d5e3923ad752d3b69f4f488f5f29667826b27aea48afba3a6d18c4
Sep 12 00:20:50.411599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.411568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvk2b" event={"ID":"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6","Type":"ContainerStarted","Data":"d42453e634d5e3923ad752d3b69f4f488f5f29667826b27aea48afba3a6d18c4"}
Sep 12 00:20:50.766903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.766818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:50.766903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.766899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:50.767451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.767415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6ae1947-3afa-47ca-913f-9744a21324f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:50.769252 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.769229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6ae1947-3afa-47ca-913f-9744a21324f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l64ss\" (UID: \"d6ae1947-3afa-47ca-913f-9744a21324f0\") " pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:50.925288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.925254    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pwzwl\""
Sep 12 00:20:50.933166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:50.933136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l64ss"
Sep 12 00:20:51.207778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:51.207749    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l64ss"]
Sep 12 00:20:51.251738 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:20:51.251706    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ae1947_3afa_47ca_913f_9744a21324f0.slice/crio-bed1ffaa32fb7de0a2bb21c8714196d2f0e6ccfafd7abb3fab675a6a348acbd8 WatchSource:0}: Error finding container bed1ffaa32fb7de0a2bb21c8714196d2f0e6ccfafd7abb3fab675a6a348acbd8: Status 404 returned error can't find the container with id bed1ffaa32fb7de0a2bb21c8714196d2f0e6ccfafd7abb3fab675a6a348acbd8
Sep 12 00:20:51.417343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:51.417316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvk2b" event={"ID":"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6","Type":"ContainerStarted","Data":"8e914f3bf6f1e17aeffb0d7e86f620239bcad96b984d2f328ed2c074baf212c3"}
Sep 12 00:20:51.421107 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:51.421076    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l64ss" event={"ID":"d6ae1947-3afa-47ca-913f-9744a21324f0","Type":"ContainerStarted","Data":"ec48c9e9e8779242c0c2bfa8b311f700b97cdd11b55ad529833aef34ed2e345a"}
Sep 12 00:20:51.421214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:51.421137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l64ss" event={"ID":"d6ae1947-3afa-47ca-913f-9744a21324f0","Type":"ContainerStarted","Data":"bed1ffaa32fb7de0a2bb21c8714196d2f0e6ccfafd7abb3fab675a6a348acbd8"}
Sep 12 00:20:52.425644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:52.425608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvk2b" event={"ID":"5d1bde87-efc7-41eb-81a5-3da1c73fb6e6","Type":"ContainerStarted","Data":"51f52585584e66785b4d0386340f2581c4c4e28b5081b8ad1c1dc1c02c4de500"}
Sep 12 00:20:52.427287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:52.427263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l64ss" event={"ID":"d6ae1947-3afa-47ca-913f-9744a21324f0","Type":"ContainerStarted","Data":"e0d6409ff5470584e6ce7dc8b1c951b9f83912a86c7261561fdf41b87810ddfc"}
Sep 12 00:20:52.440685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:52.440642    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvk2b" podStartSLOduration=252.982134494 podStartE2EDuration="4m14.440629257s" totalImagesPullingTime="1.458494763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:52.439826756 +0000 UTC m=+255.377133290" watchObservedRunningTime="2026-09-12 00:20:52.440629257 +0000 UTC m=+255.377935845"
Sep 12 00:20:53.564353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:53.564324    2562 scope.go:117] "RemoveContainer" containerID="9e67ec53a2389faf508919bc65ba41db934c4b0f7d2428fa1f02477b13381edd"
Sep 12 00:20:54.440845 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.439663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l64ss" event={"ID":"d6ae1947-3afa-47ca-913f-9744a21324f0","Type":"ContainerStarted","Data":"997942c40376bd92224d79be8cefc2aa489e60d3f6509052db86017e6675e15b"}
Sep 12 00:20:54.443359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.443333    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:20:54.443514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.443395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" event={"ID":"4947b843-449b-4830-a0fb-66cbe4077a4e","Type":"ContainerStarted","Data":"8930092ee6fb07435c97e3f1c88ff6b76a6e470f99050e4afc0c1e36e33ef7ee"}
Sep 12 00:20:54.445355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.445247    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:54.449566 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.449542    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l"
Sep 12 00:20:54.463989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.463931    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l64ss" podStartSLOduration=18.453575352 podStartE2EDuration="20.463913427s" totalImagesPullingTime="2.010338075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:20:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:54.462831416 +0000 UTC m=+257.400137937" watchObservedRunningTime="2026-09-12 00:20:54.463913427 +0000 UTC m=+257.401219948"
Sep 12 00:20:54.480536 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.480497    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-b5l7l" podStartSLOduration=22.401322043 podStartE2EDuration="25.480484273s" totalImagesPullingTime="3.07916223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:54.479270496 +0000 UTC m=+257.416577016" watchObservedRunningTime="2026-09-12 00:20:54.480484273 +0000 UTC m=+257.417790814"
Sep 12 00:20:54.495383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.495366    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:54.605342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.605299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:54.739422 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.739348    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66bc56dc8-bcvx4"]
Sep 12 00:20:54.798759 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.798717    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fc78bdd69-nrwsg"]
Sep 12 00:20:54.802335 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.802309    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.815147 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.815122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:54.821673 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.821653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc78bdd69-nrwsg"]
Sep 12 00:20:54.900180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a46fb3c0-061d-49b1-90e6-5626f7a689e3-registry-certificates\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-bound-sa-token\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-registry-tls\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a46fb3c0-061d-49b1-90e6-5626f7a689e3-ca-trust-extracted\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a46fb3c0-061d-49b1-90e6-5626f7a689e3-installation-pull-secrets\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvsp\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-kube-api-access-9cvsp\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a46fb3c0-061d-49b1-90e6-5626f7a689e3-image-registry-private-configuration\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:54.900529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:54.900397    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a46fb3c0-061d-49b1-90e6-5626f7a689e3-trusted-ca\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a46fb3c0-061d-49b1-90e6-5626f7a689e3-installation-pull-secrets\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvsp\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-kube-api-access-9cvsp\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a46fb3c0-061d-49b1-90e6-5626f7a689e3-image-registry-private-configuration\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a46fb3c0-061d-49b1-90e6-5626f7a689e3-trusted-ca\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a46fb3c0-061d-49b1-90e6-5626f7a689e3-registry-certificates\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-bound-sa-token\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-registry-tls\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.001867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.001844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a46fb3c0-061d-49b1-90e6-5626f7a689e3-ca-trust-extracted\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.002354 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.002329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a46fb3c0-061d-49b1-90e6-5626f7a689e3-ca-trust-extracted\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.002683 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.002661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a46fb3c0-061d-49b1-90e6-5626f7a689e3-registry-certificates\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.002826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.002805    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a46fb3c0-061d-49b1-90e6-5626f7a689e3-trusted-ca\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.004111 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.004079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a46fb3c0-061d-49b1-90e6-5626f7a689e3-image-registry-private-configuration\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.004301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.004280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-registry-tls\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.004377 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.004354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a46fb3c0-061d-49b1-90e6-5626f7a689e3-installation-pull-secrets\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.012830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.012811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-bound-sa-token\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.012908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.012839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvsp\" (UniqueName: \"kubernetes.io/projected/a46fb3c0-061d-49b1-90e6-5626f7a689e3-kube-api-access-9cvsp\") pod \"image-registry-fc78bdd69-nrwsg\" (UID: \"a46fb3c0-061d-49b1-90e6-5626f7a689e3\") " pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.065327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.065296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:55.112026 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.111997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:55.162854 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.162820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:55.216957 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.216930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:55.384997 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.384966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc78bdd69-nrwsg"]
Sep 12 00:20:55.390356 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.390313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:55.440544 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.440519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:55.448632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.448605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" event={"ID":"a46fb3c0-061d-49b1-90e6-5626f7a689e3","Type":"ContainerStarted","Data":"bc7a34d49340a012527f35f7c1c323adce1dd35c73a1dac1c357831b178461b0"}
Sep 12 00:20:55.448730 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:55.448660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" event={"ID":"a46fb3c0-061d-49b1-90e6-5626f7a689e3","Type":"ContainerStarted","Data":"95b45c361ba5fcf0ede0ff7dda385ae870d7b0ae35e3627f59537d4686a3509a"}
Sep 12 00:20:56.374055 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:56.374026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:56.471442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:56.471393    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podStartSLOduration=2.471377608 podStartE2EDuration="2.471377608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:20:56.470065267 +0000 UTC m=+259.407371800" watchObservedRunningTime="2026-09-12 00:20:56.471377608 +0000 UTC m=+259.408684128"
Sep 12 00:20:56.572886 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:56.572860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:57.199065 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.199032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:57.285331 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.285300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:57.339951 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.339921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:57.454737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.454667    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:20:57.480861 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.480832    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:57.785652 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.785570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:57.835260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:57.835229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:58.261485 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:58.261449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:58.320692 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:58.320653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:20:59.595352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:20:59.595317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:01.388907 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.388873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:01.553679 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.553639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:21:01.556089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.556056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6aba9d5b-aa98-4295-a190-707f83930beb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xdfwv\" (UID: \"6aba9d5b-aa98-4295-a190-707f83930beb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:21:01.566005 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.565981    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7spgb\""
Sep 12 00:21:01.574000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.573981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"
Sep 12 00:21:01.613936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.613904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:01.658299 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.658233    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w99cd"]
Sep 12 00:21:01.661775 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.661755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.664316 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.664292    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 00:21:01.664498 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.664343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 00:21:01.664498 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.664432    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j7djq\""
Sep 12 00:21:01.664614 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.664596    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 00:21:01.665256 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.665240    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 00:21:01.665325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.665249    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 00:21:01.665360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.665352    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 00:21:01.711125 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.711095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:01.754297 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-wtmp\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754297 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-root\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-tls\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-textfile\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-metrics-client-ca\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhfg\" (UniqueName: \"kubernetes.io/projected/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-kube-api-access-tfhfg\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-sys\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.754653 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.754572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.843959 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.843938    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv"]
Sep 12 00:21:01.846388 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:21:01.846363    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aba9d5b_aa98_4295_a190_707f83930beb.slice/crio-e22bdcf4b33943640382c0f270f873e58f6e1b070e35b1491729395e4e2eb3d9 WatchSource:0}: Error finding container e22bdcf4b33943640382c0f270f873e58f6e1b070e35b1491729395e4e2eb3d9: Status 404 returned error can't find the container with id e22bdcf4b33943640382c0f270f873e58f6e1b070e35b1491729395e4e2eb3d9
Sep 12 00:21:01.855033 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-textfile\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855106 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-metrics-client-ca\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855106 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhfg\" (UniqueName: \"kubernetes.io/projected/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-kube-api-access-tfhfg\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-sys\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-sys\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-wtmp\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-root\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-tls\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-wtmp\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-root\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855530 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:21:01.855434    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 12 00:21:01.855530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-textfile\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855530 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:21:01.855497    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-tls podName:d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea nodeName:}" failed. No retries permitted until 2026-09-12 00:21:02.355466908 +0000 UTC m=+265.292773418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-tls") pod "node-exporter-w99cd" (UID: "d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea") : secret "node-exporter-tls" not found
Sep 12 00:21:01.855716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-metrics-client-ca\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.855843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.855823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.857464 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.857444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.861171 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.861148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:01.862942 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.862913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhfg\" (UniqueName: \"kubernetes.io/projected/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-kube-api-access-tfhfg\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:01.961126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:01.961053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.061151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.061120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.166126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.166097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.266453 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.266382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.358724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.358691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-tls\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:02.360883 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.360860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea-node-exporter-tls\") pod \"node-exporter-w99cd\" (UID: \"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea\") " pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:02.367749 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.367728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.474435 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.474399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv" event={"ID":"6aba9d5b-aa98-4295-a190-707f83930beb","Type":"ContainerStarted","Data":"e22bdcf4b33943640382c0f270f873e58f6e1b070e35b1491729395e4e2eb3d9"}
Sep 12 00:21:02.571279 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.571245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w99cd"
Sep 12 00:21:02.583496 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:21:02.583439    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd511e8f7_5ec7_470c_b7ff_c66ed1a5f1ea.slice/crio-4420f9543935a375d719ebf525136cd92432746d3dd724bad346a633935b9cfa WatchSource:0}: Error finding container 4420f9543935a375d719ebf525136cd92432746d3dd724bad346a633935b9cfa: Status 404 returned error can't find the container with id 4420f9543935a375d719ebf525136cd92432746d3dd724bad346a633935b9cfa
Sep 12 00:21:02.621058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.621022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.711530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.711499    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:21:02.712009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.711975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.715555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.715535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.718076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718050    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 12 00:21:02.718308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718108    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 12 00:21:02.718308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718108    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 12 00:21:02.718308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718171    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 12 00:21:02.718517 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718497    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 12 00:21:02.718517 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 12 00:21:02.718631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718504    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 12 00:21:02.718631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718546    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 12 00:21:02.718631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718559    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 12 00:21:02.718631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.718542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lbbdq\""
Sep 12 00:21:02.731651 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.731626    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:21:02.822415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.822331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:02.862799 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.862799 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.862966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.863011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.863079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-out\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.863105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-web-config\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.863168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-volume\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.863193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.863251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.863241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kw65\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-kube-api-access-9kw65\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.963847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-out\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.963986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-web-config\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-volume\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kw65\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-kube-api-access-9kw65\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.964449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.964224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.965128 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.965099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.966271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.966241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.967224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.967181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.967320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.967297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-out\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.967695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.967647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-volume\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.967889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.967866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-web-config\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.967965 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.967941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.968455 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.968430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.969066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.969040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.969372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.969350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.969713 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.969687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:02.972041 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:02.972020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kw65\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-kube-api-access-9kw65\") pod \"alertmanager-main-0\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:03.025002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.024956    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:21:03.401942 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.401857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:21:03.404734 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:21:03.404710    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594e44f4_7285_4eb6_870b_ffbbae6f709b.slice/crio-6eef8052c03be97de20fdfed536e0d28b5ed5f3e24cb17bd205d95effbd1d8db WatchSource:0}: Error finding container 6eef8052c03be97de20fdfed536e0d28b5ed5f3e24cb17bd205d95effbd1d8db: Status 404 returned error can't find the container with id 6eef8052c03be97de20fdfed536e0d28b5ed5f3e24cb17bd205d95effbd1d8db
Sep 12 00:21:03.479648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.479614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv" event={"ID":"6aba9d5b-aa98-4295-a190-707f83930beb","Type":"ContainerStarted","Data":"567f89e11a6e7cf6e94a4b41bfdaba4a965eb0b14fad32377b23db8fe2e9f847"}
Sep 12 00:21:03.480892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.480864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99cd" event={"ID":"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea","Type":"ContainerStarted","Data":"4420f9543935a375d719ebf525136cd92432746d3dd724bad346a633935b9cfa"}
Sep 12 00:21:03.482000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.481978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"6eef8052c03be97de20fdfed536e0d28b5ed5f3e24cb17bd205d95effbd1d8db"}
Sep 12 00:21:03.495364 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.495322    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xdfwv" podStartSLOduration=33.163056355 podStartE2EDuration="34.495309216s" totalImagesPullingTime="1.332252861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:21:03.495085507 +0000 UTC m=+266.432392029" watchObservedRunningTime="2026-09-12 00:21:03.495309216 +0000 UTC m=+266.432615735"
Sep 12 00:21:03.620827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.620799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:03.726333 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:03.726264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:04.609704 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:04.609668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:04.709151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:04.709122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:04.745158 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:04.745126    2562 patch_prober.go:28] interesting pod/image-registry-66bc56dc8-bcvx4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:04.745328 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:04.745185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:04.808849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:04.808816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:05.030963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:05.030885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:06.407824 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.407790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:06.462766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.462737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:06.495124 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.495097    2562 generic.go:356] "Generic (PLEG): container finished" podID="d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea" containerID="b33f1e564e77b6985826bc573bd43fa02d9ed9b7d2ff00f7e35b6b4f2044a96f" exitCode=0
Sep 12 00:21:06.495255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.495177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99cd" event={"ID":"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea","Type":"ContainerDied","Data":"b33f1e564e77b6985826bc573bd43fa02d9ed9b7d2ff00f7e35b6b4f2044a96f"}
Sep 12 00:21:06.497008 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.496977    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074" exitCode=0
Sep 12 00:21:06.497105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.497019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"}
Sep 12 00:21:06.973622 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:06.973591    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:07.502249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:07.502214    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99cd" event={"ID":"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea","Type":"ContainerStarted","Data":"c8325b3ff58d3f519188aa6796ad77d35a8648e4f80d26b032b6da5fca9c44b4"}
Sep 12 00:21:07.502636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:07.502263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w99cd" event={"ID":"d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea","Type":"ContainerStarted","Data":"4d4b91511b5d0a8fcfd85f4997b83bc88dceba854ecd592739fd37d6f0be8579"}
Sep 12 00:21:07.523164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:07.523115    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w99cd" podStartSLOduration=3.655070272 podStartE2EDuration="6.523095204s" totalImagesPullingTime="2.805171265s" totalInitContainerRuntime="62.853667ms" isStatefulPod=true podCreationTimestamp="2026-09-12 00:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:21:07.521688445 +0000 UTC m=+270.458994966" watchObservedRunningTime="2026-09-12 00:21:07.523095204 +0000 UTC m=+270.460401725"
Sep 12 00:21:07.939492 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:07.939446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:08.986124 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:08.986087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:09.043352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.043320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:09.513644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.513617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"}
Sep 12 00:21:09.513859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.513676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"}
Sep 12 00:21:09.513859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.513703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"}
Sep 12 00:21:09.513859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.513729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"}
Sep 12 00:21:09.513859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.513752    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"}
Sep 12 00:21:09.903778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.903750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:09.951726 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:09.951701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:10.523004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:10.522972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerStarted","Data":"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"}
Sep 12 00:21:10.551678 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:10.551616    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.686823816 podStartE2EDuration="8.551595625s" totalImagesPullingTime="4.848259797s" totalInitContainerRuntime="16.512012ms" isStatefulPod=false podCreationTimestamp="2026-09-12 00:21:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:21:10.549974639 +0000 UTC m=+273.487281159" watchObservedRunningTime="2026-09-12 00:21:10.551595625 +0000 UTC m=+273.488902146"
Sep 12 00:21:10.788060 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:10.787975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:10.835341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:10.835308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:14.587135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.587098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5757bd6795-hhp5k"]
Sep 12 00:21:14.589943 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.589918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.592314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.592296    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 12 00:21:14.593376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593355    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 12 00:21:14.593466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593371    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 12 00:21:14.593466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593429    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 12 00:21:14.593599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 12 00:21:14.593599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bsxwp\""
Sep 12 00:21:14.593599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 12 00:21:14.593830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.593814    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 12 00:21:14.597975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.597957    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 12 00:21:14.602282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.602263    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5757bd6795-hhp5k"]
Sep 12 00:21:14.743948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.743917    2562 patch_prober.go:28] interesting pod/image-registry-66bc56dc8-bcvx4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:14.744114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.743976    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:14.763523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-trusted-ca-bundle\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.763618 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhm6\" (UniqueName: \"kubernetes.io/projected/f98b40aa-e821-4ac1-9231-4be9334e370f-kube-api-access-fhhm6\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.763618 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-serving-cert\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.763618 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763610    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-console-config\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.763728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-oauth-config\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.763728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-service-ca\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.763728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.763699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-oauth-serving-cert\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-trusted-ca-bundle\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhm6\" (UniqueName: \"kubernetes.io/projected/f98b40aa-e821-4ac1-9231-4be9334e370f-kube-api-access-fhhm6\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-serving-cert\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864430 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-console-config\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864430 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-oauth-config\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864430 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-service-ca\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864430 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-oauth-serving-cert\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.864977 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.864954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-console-config\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.865088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.865073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-trusted-ca-bundle\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.865152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.865131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-oauth-serving-cert\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.865194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.865140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-service-ca\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.866556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.866532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-oauth-config\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.866736 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.866720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-serving-cert\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.871602 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.871583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhm6\" (UniqueName: \"kubernetes.io/projected/f98b40aa-e821-4ac1-9231-4be9334e370f-kube-api-access-fhhm6\") pod \"console-5757bd6795-hhp5k\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") " pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.898704 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.898666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:14.990359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:14.990327    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:15.039577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:15.039551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:15.116127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:15.116050    2562 patch_prober.go:28] interesting pod/image-registry-fc78bdd69-nrwsg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:15.116127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:15.116103    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:15.168940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:15.168914    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5757bd6795-hhp5k"]
Sep 12 00:21:15.171200 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:21:15.171172    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98b40aa_e821_4ac1_9231_4be9334e370f.slice/crio-9b760bc5621a8700509bbf5f47904fb15932cd44205e1dc5f986e082c1d41498 WatchSource:0}: Error finding container 9b760bc5621a8700509bbf5f47904fb15932cd44205e1dc5f986e082c1d41498: Status 404 returned error can't find the container with id 9b760bc5621a8700509bbf5f47904fb15932cd44205e1dc5f986e082c1d41498
Sep 12 00:21:15.540539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:15.540442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5757bd6795-hhp5k" event={"ID":"f98b40aa-e821-4ac1-9231-4be9334e370f","Type":"ContainerStarted","Data":"9b760bc5621a8700509bbf5f47904fb15932cd44205e1dc5f986e082c1d41498"}
Sep 12 00:21:17.103682 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:17.103646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:17.179234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:17.179197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:17.231459 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:17.231427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:18.131097 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:21:18.131020    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cpf2k" podUID="9ba59c17-33b2-4642-a51c-e92070a474d0"
Sep 12 00:21:18.462095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:18.462012    2562 patch_prober.go:28] interesting pod/image-registry-fc78bdd69-nrwsg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:18.462095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:18.462066    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:18.553734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:18.553698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5757bd6795-hhp5k" event={"ID":"f98b40aa-e821-4ac1-9231-4be9334e370f","Type":"ContainerStarted","Data":"aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911"}
Sep 12 00:21:18.553886 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:18.553776    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:21:18.571049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:18.571002    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5757bd6795-hhp5k" podStartSLOduration=1.921924485 podStartE2EDuration="4.570987927s" totalImagesPullingTime="2.649063442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:21:18.568824818 +0000 UTC m=+281.506131329" watchObservedRunningTime="2026-09-12 00:21:18.570987927 +0000 UTC m=+281.508294446"
Sep 12 00:21:19.758512 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:19.758440    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerName="registry" containerID="cri-o://bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a" gracePeriod=30
Sep 12 00:21:19.996441 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:19.996419    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:21:20.006875 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.006855    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-installation-pull-secrets\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.006963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.006892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-image-registry-private-configuration\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.006963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.006914    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.006963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.006929    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-bound-sa-token\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.006963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.006958    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-ca-trust-extracted\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.007167 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.006985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-trusted-ca\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.007167 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.007005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv678\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-kube-api-access-cv678\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.007167 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.007022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-certificates\") pod \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\" (UID: \"0c42d4f0-6198-4b4e-a453-e94a40cadfdf\") "
Sep 12 00:21:20.007556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.007528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:21:20.007556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.007541    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:21:20.009486 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.009390    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:21:20.009655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.009627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:21:20.009742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.009636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:21:20.009742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.009677    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:21:20.009850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.009826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-kube-api-access-cv678" (OuterVolumeSpecName: "kube-api-access-cv678") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "kube-api-access-cv678". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:21:20.017865 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.017838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0c42d4f0-6198-4b4e-a453-e94a40cadfdf" (UID: "0c42d4f0-6198-4b4e-a453-e94a40cadfdf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:21:20.108326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108297    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108321    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-bound-sa-token\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108330    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-ca-trust-extracted\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108339    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-trusted-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108348    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv678\" (UniqueName: \"kubernetes.io/projected/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-kube-api-access-cv678\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108355    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-registry-certificates\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108364    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-installation-pull-secrets\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.108546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.108373    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c42d4f0-6198-4b4e-a453-e94a40cadfdf-image-registry-private-configuration\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:21:20.561589 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.561555    2562 generic.go:356] "Generic (PLEG): container finished" podID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerID="bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a" exitCode=0
Sep 12 00:21:20.561746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.561610    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4"
Sep 12 00:21:20.561746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.561630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" event={"ID":"0c42d4f0-6198-4b4e-a453-e94a40cadfdf","Type":"ContainerDied","Data":"bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a"}
Sep 12 00:21:20.561746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.561698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bc56dc8-bcvx4" event={"ID":"0c42d4f0-6198-4b4e-a453-e94a40cadfdf","Type":"ContainerDied","Data":"25e38f76222b3fd63b92d29750320bfddb6d4737eabab2b544f263f1428a8603"}
Sep 12 00:21:20.561746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.561719    2562 scope.go:117] "RemoveContainer" containerID="bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a"
Sep 12 00:21:20.570664 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.570647    2562 scope.go:117] "RemoveContainer" containerID="bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a"
Sep 12 00:21:20.570983 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:21:20.570964    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a\": container with ID starting with bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a not found: ID does not exist" containerID="bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a"
Sep 12 00:21:20.571042 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.570989    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a"} err="failed to get container status \"bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a\": rpc error: code = NotFound desc = could not find container \"bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a\": container with ID starting with bcba62a064f01d3b4eb40ad569c20af36a628b0ece0faa4547ea0d1f7dc7a64a not found: ID does not exist"
Sep 12 00:21:20.582657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.582634    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66bc56dc8-bcvx4"]
Sep 12 00:21:20.586387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:20.586367    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66bc56dc8-bcvx4"]
Sep 12 00:21:21.317510 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:21.317463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:21:21.319672 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:21.319652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ba59c17-33b2-4642-a51c-e92070a474d0-metrics-tls\") pod \"dns-default-cpf2k\" (UID: \"9ba59c17-33b2-4642-a51c-e92070a474d0\") " pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:21:21.557312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:21.557280    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-knhwr\""
Sep 12 00:21:21.565227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:21.565202    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:21:21.569084 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:21.569025    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" path="/var/lib/kubelet/pods/0c42d4f0-6198-4b4e-a453-e94a40cadfdf/volumes"
Sep 12 00:21:21.835287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:21.835264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cpf2k"]
Sep 12 00:21:21.837414 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:21:21.837388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba59c17_33b2_4642_a51c_e92070a474d0.slice/crio-dda0345708318512fed2be26c056e2453d2cb1ed1f34dade5653b132abb6b288 WatchSource:0}: Error finding container dda0345708318512fed2be26c056e2453d2cb1ed1f34dade5653b132abb6b288: Status 404 returned error can't find the container with id dda0345708318512fed2be26c056e2453d2cb1ed1f34dade5653b132abb6b288
Sep 12 00:21:22.572593 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:22.572558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpf2k" event={"ID":"9ba59c17-33b2-4642-a51c-e92070a474d0","Type":"ContainerStarted","Data":"dda0345708318512fed2be26c056e2453d2cb1ed1f34dade5653b132abb6b288"}
Sep 12 00:21:24.580516 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.580484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpf2k" event={"ID":"9ba59c17-33b2-4642-a51c-e92070a474d0","Type":"ContainerStarted","Data":"fd0c87c7929919057832b5c68dfd1380cdffc3c7330018f40627990785f13ea3"}
Sep 12 00:21:24.580890 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.580534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpf2k" event={"ID":"9ba59c17-33b2-4642-a51c-e92070a474d0","Type":"ContainerStarted","Data":"ee1de81f7d25b6dfd789e75b60bb010d7223bc0879566184bfb6fe74b830c764"}
Sep 12 00:21:24.580890 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.580619    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:21:24.596720 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.596676    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cpf2k" podStartSLOduration=251.808334963 podStartE2EDuration="4m13.59666185s" totalImagesPullingTime="1.788326887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:21:24.595037866 +0000 UTC m=+287.532344398" watchObservedRunningTime="2026-09-12 00:21:24.59666185 +0000 UTC m=+287.533968385"
Sep 12 00:21:24.899174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.899090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:24.899174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.899165    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:24.900585 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.900566    2562 patch_prober.go:28] interesting pod/console-5757bd6795-hhp5k container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" start-of-body=
Sep 12 00:21:24.900640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:24.900605    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5757bd6795-hhp5k" podUID="f98b40aa-e821-4ac1-9231-4be9334e370f" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused"
Sep 12 00:21:25.116076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:25.116044    2562 patch_prober.go:28] interesting pod/image-registry-fc78bdd69-nrwsg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:25.116213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:25.116098    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:28.402064 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:28.402031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:28.452431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:28.452405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:28.462009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:28.461985    2562 patch_prober.go:28] interesting pod/image-registry-fc78bdd69-nrwsg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:28.462086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:28.462029    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:31.058575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:31.058536    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:31.120528 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:31.120495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:34.585603 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:34.585570    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cpf2k"
Sep 12 00:21:34.874638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:34.874565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:34.903609 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:34.903583    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:34.907277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:34.907251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:21:35.116467 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:35.116432    2562 patch_prober.go:28] interesting pod/image-registry-fc78bdd69-nrwsg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:35.116644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:35.116523    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:35.116644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:35.116561    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:21:35.117068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:35.117050    2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"bc7a34d49340a012527f35f7c1c323adce1dd35c73a1dac1c357831b178461b0"} pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 12 00:21:35.120750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:35.120720    2562 patch_prober.go:28] interesting pod/image-registry-fc78bdd69-nrwsg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 00:21:35.120878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:35.120760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:21:37.026738 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:37.026708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:37.422366 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:37.422003    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:21:37.422366 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:37.422187    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:21:37.441582 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:37.441562    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 00:21:39.901351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:39.901318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:39.957875 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:39.957843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:40.123387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:40.123356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:40.224794 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:40.224694    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:45.120789 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:45.120758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:21:45.239441 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:45.239410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:45.438613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:45.438525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:45.539823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:45.539790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:54.277451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.277416    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:54.330928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.330899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:54.688657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.688629    2562 generic.go:356] "Generic (PLEG): container finished" podID="87f8b85f-b68a-47c2-abbd-091e63c3f3d1" containerID="ecfb3ba0248bf0c7c3287ce4f6dd9b07952b54e6ada0d371dbb48afb798defc3" exitCode=0
Sep 12 00:21:54.688787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.688700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv" event={"ID":"87f8b85f-b68a-47c2-abbd-091e63c3f3d1","Type":"ContainerDied","Data":"ecfb3ba0248bf0c7c3287ce4f6dd9b07952b54e6ada0d371dbb48afb798defc3"}
Sep 12 00:21:54.689101 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.689081    2562 scope.go:117] "RemoveContainer" containerID="ecfb3ba0248bf0c7c3287ce4f6dd9b07952b54e6ada0d371dbb48afb798defc3"
Sep 12 00:21:54.908613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.908537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:54.965486 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:54.965451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:21:55.694227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:55.694195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bjdzv" event={"ID":"87f8b85f-b68a-47c2-abbd-091e63c3f3d1","Type":"ContainerStarted","Data":"ea84d214afbb015342bbcccdc70bea3cc7506612749f1ee8c7429ff01e2f45f7"}
Sep 12 00:21:59.709831 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:59.709796    2562 generic.go:356] "Generic (PLEG): container finished" podID="b3562c34-1caf-42eb-9b2a-56abc7092a19" containerID="787cf0d67d8c395bcdca0ee02f61205c3c2dc8180595a92a0640253e7997016d" exitCode=0
Sep 12 00:21:59.710232 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:59.709841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qd42h" event={"ID":"b3562c34-1caf-42eb-9b2a-56abc7092a19","Type":"ContainerDied","Data":"787cf0d67d8c395bcdca0ee02f61205c3c2dc8180595a92a0640253e7997016d"}
Sep 12 00:21:59.710232 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:21:59.710158    2562 scope.go:117] "RemoveContainer" containerID="787cf0d67d8c395bcdca0ee02f61205c3c2dc8180595a92a0640253e7997016d"
Sep 12 00:22:00.135397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:00.135356    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" podUID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerName="registry" containerID="cri-o://bc7a34d49340a012527f35f7c1c323adce1dd35c73a1dac1c357831b178461b0" gracePeriod=30
Sep 12 00:22:00.600903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:00.600849    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5hxx_9e8227bd-18e8-4220-82fd-2e9878a7aecb/serve-healthcheck-canary/0.log"
Sep 12 00:22:00.715281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:00.715245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qd42h" event={"ID":"b3562c34-1caf-42eb-9b2a-56abc7092a19","Type":"ContainerStarted","Data":"bf80c99057a758ec7b1d56bb88ac261e4e2a99dfe147fa531d5b21c0d592fad5"}
Sep 12 00:22:01.720836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:01.720795    2562 generic.go:356] "Generic (PLEG): container finished" podID="a46fb3c0-061d-49b1-90e6-5626f7a689e3" containerID="bc7a34d49340a012527f35f7c1c323adce1dd35c73a1dac1c357831b178461b0" exitCode=0
Sep 12 00:22:01.721241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:01.720851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" event={"ID":"a46fb3c0-061d-49b1-90e6-5626f7a689e3","Type":"ContainerDied","Data":"bc7a34d49340a012527f35f7c1c323adce1dd35c73a1dac1c357831b178461b0"}
Sep 12 00:22:01.721241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:01.720893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg" event={"ID":"a46fb3c0-061d-49b1-90e6-5626f7a689e3","Type":"ContainerStarted","Data":"de04cae0ae3f5fb503d7502ad37921ea1e5a091a6f4c749998844865043b5b8b"}
Sep 12 00:22:01.721241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:01.720918    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:22:01.987549 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:01.987450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:02.041579 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:02.041545    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:09.909914 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:09.909880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:09.962223 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:09.962195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:12.823347 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:12.823308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:17.122653 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:17.122621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:17.222858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:17.222830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:17.276219 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:17.276191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:20.871341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:20.871308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:21.112670 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.112636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:21.936572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936537    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:22:21.936985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936826    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="alertmanager" containerID="cri-o://929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314" gracePeriod=120
Sep 12 00:22:21.936985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936906    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-metric" containerID="cri-o://0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd" gracePeriod=120
Sep 12 00:22:21.936985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936923    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-web" containerID="cri-o://f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992" gracePeriod=120
Sep 12 00:22:21.937159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936955    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="prom-label-proxy" containerID="cri-o://b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3" gracePeriod=120
Sep 12 00:22:21.937159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936974    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="config-reloader" containerID="cri-o://bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136" gracePeriod=120
Sep 12 00:22:21.937159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:21.936997    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy" containerID="cri-o://c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f" gracePeriod=120
Sep 12 00:22:22.729972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.729941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fc78bdd69-nrwsg"
Sep 12 00:22:22.807991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.807959    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3" exitCode=0
Sep 12 00:22:22.807991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.807986    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f" exitCode=0
Sep 12 00:22:22.807991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.807995    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136" exitCode=0
Sep 12 00:22:22.807991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.808001    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314" exitCode=0
Sep 12 00:22:22.808295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.808029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"}
Sep 12 00:22:22.808295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.808105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"}
Sep 12 00:22:22.808295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.808147    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"}
Sep 12 00:22:22.808295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:22.808182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"}
Sep 12 00:22:23.008281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.008189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:23.058572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.058544    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:23.183362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.183338    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.208806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208775    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208815    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-web-config\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208850    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-main-tls\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-main-db\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208917    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-metrics-client-ca\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.208984    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-out\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209011    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-cluster-tls-config\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-web\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209083    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-tls-assets\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209107    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-volume\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209138    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-trusted-ca-bundle\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.209451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209170    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kw65\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-kube-api-access-9kw65\") pod \"594e44f4-7285-4eb6-870b-ffbbae6f709b\" (UID: \"594e44f4-7285-4eb6-870b-ffbbae6f709b\") "
Sep 12 00:22:23.210421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.209929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:22:23.210421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.210098    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:22:23.210611 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.210549    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:22:23.212465 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.212417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-kube-api-access-9kw65" (OuterVolumeSpecName: "kube-api-access-9kw65") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "kube-api-access-9kw65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:22:23.212604 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.212543    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.212704 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.212576    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.213689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.213651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.213774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.213559    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.214222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.214185    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:22:23.215146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.215109    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-volume" (OuterVolumeSpecName: "config-volume") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.216114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.216088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-out" (OuterVolumeSpecName: "config-out") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:22:23.218975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.218877    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.225572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.225546    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-web-config" (OuterVolumeSpecName: "web-config") pod "594e44f4-7285-4eb6-870b-ffbbae6f709b" (UID: "594e44f4-7285-4eb6-870b-ffbbae6f709b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310045    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310080    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kw65\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-kube-api-access-9kw65\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310090    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310099    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-web-config\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310108    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-main-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310118    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-alertmanager-main-db\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310126    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/594e44f4-7285-4eb6-870b-ffbbae6f709b-metrics-client-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310135    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310145    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-out\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310153    2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-cluster-tls-config\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310162    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310170    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/594e44f4-7285-4eb6-870b-ffbbae6f709b-tls-assets\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.310506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.310178    2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/594e44f4-7285-4eb6-870b-ffbbae6f709b-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:22:23.815294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815264    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd" exitCode=0
Sep 12 00:22:23.815294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815287    2562 generic.go:356] "Generic (PLEG): container finished" podID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerID="f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992" exitCode=0
Sep 12 00:22:23.815489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"}
Sep 12 00:22:23.815489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815349    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.815489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"}
Sep 12 00:22:23.815489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"594e44f4-7285-4eb6-870b-ffbbae6f709b","Type":"ContainerDied","Data":"6eef8052c03be97de20fdfed536e0d28b5ed5f3e24cb17bd205d95effbd1d8db"}
Sep 12 00:22:23.815636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.815495    2562 scope.go:117] "RemoveContainer" containerID="b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"
Sep 12 00:22:23.823119 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.823101    2562 scope.go:117] "RemoveContainer" containerID="0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"
Sep 12 00:22:23.829953 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.829934    2562 scope.go:117] "RemoveContainer" containerID="c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"
Sep 12 00:22:23.839862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.839768    2562 scope.go:117] "RemoveContainer" containerID="f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"
Sep 12 00:22:23.840159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.840111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:22:23.842034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.842013    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:22:23.846971 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.846955    2562 scope.go:117] "RemoveContainer" containerID="bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"
Sep 12 00:22:23.853717 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.853694    2562 scope.go:117] "RemoveContainer" containerID="929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"
Sep 12 00:22:23.860603 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.860581    2562 scope.go:117] "RemoveContainer" containerID="d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"
Sep 12 00:22:23.862376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862359    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:22:23.862642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862630    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862644    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862653    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-web"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862659    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-web"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862673    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="alertmanager"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862678    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="alertmanager"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862686    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerName="registry"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862691    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerName="registry"
Sep 12 00:22:23.862696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862697    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="prom-label-proxy"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862703    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="prom-label-proxy"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862709    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-metric"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862714    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-metric"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862720    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="config-reloader"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862724    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="config-reloader"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862730    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="init-config-reloader"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862735    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="init-config-reloader"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862778    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="prom-label-proxy"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862786    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-metric"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862792    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c42d4f0-6198-4b4e-a453-e94a40cadfdf" containerName="registry"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862798    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="config-reloader"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862803    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="alertmanager"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862809    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy-web"
Sep 12 00:22:23.862970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.862815    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" containerName="kube-rbac-proxy"
Sep 12 00:22:23.867553 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.867536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.867720 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.867701    2562 scope.go:117] "RemoveContainer" containerID="b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"
Sep 12 00:22:23.868026 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.867994    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3\": container with ID starting with b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3 not found: ID does not exist" containerID="b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"
Sep 12 00:22:23.868094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868037    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"} err="failed to get container status \"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3\": rpc error: code = NotFound desc = could not find container \"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3\": container with ID starting with b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3 not found: ID does not exist"
Sep 12 00:22:23.868094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868063    2562 scope.go:117] "RemoveContainer" containerID="0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"
Sep 12 00:22:23.868363 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.868341    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd\": container with ID starting with 0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd not found: ID does not exist" containerID="0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"
Sep 12 00:22:23.868490 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868369    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"} err="failed to get container status \"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd\": rpc error: code = NotFound desc = could not find container \"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd\": container with ID starting with 0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd not found: ID does not exist"
Sep 12 00:22:23.868490 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868385    2562 scope.go:117] "RemoveContainer" containerID="c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"
Sep 12 00:22:23.868646 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.868629    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f\": container with ID starting with c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f not found: ID does not exist" containerID="c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"
Sep 12 00:22:23.868689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868651    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"} err="failed to get container status \"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f\": rpc error: code = NotFound desc = could not find container \"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f\": container with ID starting with c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f not found: ID does not exist"
Sep 12 00:22:23.868689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868665    2562 scope.go:117] "RemoveContainer" containerID="f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"
Sep 12 00:22:23.868917 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.868897    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992\": container with ID starting with f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992 not found: ID does not exist" containerID="f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"
Sep 12 00:22:23.868968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868923    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"} err="failed to get container status \"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992\": rpc error: code = NotFound desc = could not find container \"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992\": container with ID starting with f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992 not found: ID does not exist"
Sep 12 00:22:23.868968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.868935    2562 scope.go:117] "RemoveContainer" containerID="bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"
Sep 12 00:22:23.869185 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.869168    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136\": container with ID starting with bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136 not found: ID does not exist" containerID="bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"
Sep 12 00:22:23.869224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869187    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"} err="failed to get container status \"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136\": rpc error: code = NotFound desc = could not find container \"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136\": container with ID starting with bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136 not found: ID does not exist"
Sep 12 00:22:23.869224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869197    2562 scope.go:117] "RemoveContainer" containerID="929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"
Sep 12 00:22:23.869439 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.869422    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314\": container with ID starting with 929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314 not found: ID does not exist" containerID="929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"
Sep 12 00:22:23.869514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869443    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"} err="failed to get container status \"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314\": rpc error: code = NotFound desc = could not find container \"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314\": container with ID starting with 929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314 not found: ID does not exist"
Sep 12 00:22:23.869514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869458    2562 scope.go:117] "RemoveContainer" containerID="d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"
Sep 12 00:22:23.869745 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:22:23.869723    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074\": container with ID starting with d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074 not found: ID does not exist" containerID="d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"
Sep 12 00:22:23.869783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869753    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"} err="failed to get container status \"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074\": rpc error: code = NotFound desc = could not find container \"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074\": container with ID starting with d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074 not found: ID does not exist"
Sep 12 00:22:23.869783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869772    2562 scope.go:117] "RemoveContainer" containerID="b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"
Sep 12 00:22:23.869850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869784    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 12 00:22:23.869850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.869784    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 12 00:22:23.870021 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870004    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3"} err="failed to get container status \"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3\": rpc error: code = NotFound desc = could not find container \"b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3\": container with ID starting with b75b2e7080fdb433e1214e454879ff0041c195a67ec8cce5be95ce9e1022fab3 not found: ID does not exist"
Sep 12 00:22:23.870063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870027    2562 scope.go:117] "RemoveContainer" containerID="0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"
Sep 12 00:22:23.870204 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870187    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 12 00:22:23.870255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870187    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 12 00:22:23.870320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870297    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd"} err="failed to get container status \"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd\": rpc error: code = NotFound desc = could not find container \"0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd\": container with ID starting with 0ae28974a8fdb520c238b5056323bb593d1ce87feba27d19ddccc2416fa3b3fd not found: ID does not exist"
Sep 12 00:22:23.870367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870322    2562 scope.go:117] "RemoveContainer" containerID="c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"
Sep 12 00:22:23.870416 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870401    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 12 00:22:23.870534 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 12 00:22:23.870644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870599    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f"} err="failed to get container status \"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f\": rpc error: code = NotFound desc = could not find container \"c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f\": container with ID starting with c21923eede2c3a161770b6cf8d9a2ca98920eef912cfa88c6f0619639c18537f not found: ID does not exist"
Sep 12 00:22:23.870644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870631    2562 scope.go:117] "RemoveContainer" containerID="f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"
Sep 12 00:22:23.870754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870641    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 12 00:22:23.870754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870669    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lbbdq\""
Sep 12 00:22:23.870934 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870901    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992"} err="failed to get container status \"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992\": rpc error: code = NotFound desc = could not find container \"f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992\": container with ID starting with f5ddb05e2f997e8d5f3ee46a4ef28f67419f43585b5179c8e8e189cad2a28992 not found: ID does not exist"
Sep 12 00:22:23.871000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.870935    2562 scope.go:117] "RemoveContainer" containerID="bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"
Sep 12 00:22:23.871062 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.871010    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 12 00:22:23.871217 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.871192    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136"} err="failed to get container status \"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136\": rpc error: code = NotFound desc = could not find container \"bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136\": container with ID starting with bfac3e3b6cd0d6812aca765f284ea35945f0f86037ffc5632f008131d0c62136 not found: ID does not exist"
Sep 12 00:22:23.871272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.871220    2562 scope.go:117] "RemoveContainer" containerID="929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"
Sep 12 00:22:23.871533 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.871512    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314"} err="failed to get container status \"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314\": rpc error: code = NotFound desc = could not find container \"929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314\": container with ID starting with 929e9e704e2fa1df363334318072385e4f713582df40dab592c0572f0029b314 not found: ID does not exist"
Sep 12 00:22:23.871533 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.871531    2562 scope.go:117] "RemoveContainer" containerID="d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"
Sep 12 00:22:23.871894 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.871869    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074"} err="failed to get container status \"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074\": rpc error: code = NotFound desc = could not find container \"d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074\": container with ID starting with d5deaf122cc868451c504bea9752c5b5c77f920b45273d899def828e21554074 not found: ID does not exist"
Sep 12 00:22:23.874898 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.874747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 12 00:22:23.883119 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.883098    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:22:23.914489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-web-config\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-config-volume\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91689675-72b3-4b55-a95b-675c191cf870-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91689675-72b3-4b55-a95b-675c191cf870-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772hk\" (UniqueName: \"kubernetes.io/projected/91689675-72b3-4b55-a95b-675c191cf870-kube-api-access-772hk\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91689675-72b3-4b55-a95b-675c191cf870-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91689675-72b3-4b55-a95b-675c191cf870-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914759    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91689675-72b3-4b55-a95b-675c191cf870-config-out\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:23.914826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:23.914791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.015973 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.015940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.015973 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.015974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-web-config\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-config-volume\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91689675-72b3-4b55-a95b-675c191cf870-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91689675-72b3-4b55-a95b-675c191cf870-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772hk\" (UniqueName: \"kubernetes.io/projected/91689675-72b3-4b55-a95b-675c191cf870-kube-api-access-772hk\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91689675-72b3-4b55-a95b-675c191cf870-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91689675-72b3-4b55-a95b-675c191cf870-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91689675-72b3-4b55-a95b-675c191cf870-config-out\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.016539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.016460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91689675-72b3-4b55-a95b-675c191cf870-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.017166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.017027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91689675-72b3-4b55-a95b-675c191cf870-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.017974 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.017632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91689675-72b3-4b55-a95b-675c191cf870-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.019188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.019099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.019288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.019208    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.019288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.019262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-config-volume\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.019415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.019403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-web-config\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.019489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.019434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.019765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.019743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91689675-72b3-4b55-a95b-675c191cf870-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.020498 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.020450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.020609 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.020588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91689675-72b3-4b55-a95b-675c191cf870-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.021194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.021179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91689675-72b3-4b55-a95b-675c191cf870-config-out\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.024024 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.023997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772hk\" (UniqueName: \"kubernetes.io/projected/91689675-72b3-4b55-a95b-675c191cf870-kube-api-access-772hk\") pod \"alertmanager-main-0\" (UID: \"91689675-72b3-4b55-a95b-675c191cf870\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.177520 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.177390    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 12 00:22:24.456543 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.456513    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 12 00:22:24.458183 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:22:24.458154    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91689675_72b3_4b55_a95b_675c191cf870.slice/crio-e0fa3eb167ef55ecd2f3d965392396d8abccf6ec20a4b9bd1805733475ef15cf WatchSource:0}: Error finding container e0fa3eb167ef55ecd2f3d965392396d8abccf6ec20a4b9bd1805733475ef15cf: Status 404 returned error can't find the container with id e0fa3eb167ef55ecd2f3d965392396d8abccf6ec20a4b9bd1805733475ef15cf
Sep 12 00:22:24.823034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.822999    2562 generic.go:356] "Generic (PLEG): container finished" podID="91689675-72b3-4b55-a95b-675c191cf870" containerID="e78feb4c7358f2178e150bf8b3427bfad12d21a5c64b630d758a21a124602380" exitCode=0
Sep 12 00:22:24.823177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.823078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerDied","Data":"e78feb4c7358f2178e150bf8b3427bfad12d21a5c64b630d758a21a124602380"}
Sep 12 00:22:24.823177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:24.823150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"e0fa3eb167ef55ecd2f3d965392396d8abccf6ec20a4b9bd1805733475ef15cf"}
Sep 12 00:22:25.569009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.568975    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594e44f4-7285-4eb6-870b-ffbbae6f709b" path="/var/lib/kubelet/pods/594e44f4-7285-4eb6-870b-ffbbae6f709b/volumes"
Sep 12 00:22:25.830103 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.830010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"a45c4746c6b1a14405bee8ca257da2f55d11e2cefba9aa4dfc0323e6fd0e79ac"}
Sep 12 00:22:25.830272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.830121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"74fed81b8951fb34cee825eb2998d3ac1ec3a4adb2436e64a565ed1e80db6279"}
Sep 12 00:22:25.830272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.830161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"0d8424bfe46afebc3efadd0a092b52a1cd5e0d9f973740797948bfd1d4eaa55c"}
Sep 12 00:22:25.830272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.830201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"3a9a95c012ac8de777abe6494488e9ad992aec0aab17410a5f78ec517d7c425b"}
Sep 12 00:22:25.830272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.830236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"af43ebd18de0420af16a213bd60bd67256bc5621df7255350dfeb7d2217cfb37"}
Sep 12 00:22:25.830272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.830270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91689675-72b3-4b55-a95b-675c191cf870","Type":"ContainerStarted","Data":"1e946ba8bc7879cb9980602be0cd85e7d6c2deb1c9b2b6817ac9971dd507ab45"}
Sep 12 00:22:25.859879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.859824    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.844018217 podStartE2EDuration="2.859808897s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.79068ms" isStatefulPod=false podCreationTimestamp="2026-09-12 00:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:22:25.858975762 +0000 UTC m=+348.796282281" watchObservedRunningTime="2026-09-12 00:22:25.859808897 +0000 UTC m=+348.797115416"
Sep 12 00:22:25.956436 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.956403    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-589dd97556-tc7hb"]
Sep 12 00:22:25.963252 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.963221    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:25.965900 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.965877    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-56qmk\""
Sep 12 00:22:25.966019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.965976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 12 00:22:25.966209 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.966192    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 12 00:22:25.966273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.966256    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 12 00:22:25.966338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.966285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 12 00:22:25.966338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.966296    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 12 00:22:25.971513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.971272    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 12 00:22:25.973108 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:25.973082    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-589dd97556-tc7hb"]
Sep 12 00:22:26.032767 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-telemeter-trusted-ca-bundle\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.032767 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.032989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-serving-certs-ca-bundle\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.032989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-telemeter-client-tls\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.032989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5fn\" (UniqueName: \"kubernetes.io/projected/94edbbbf-ee22-4f2e-9444-634d7faec842-kube-api-access-8s5fn\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.032989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-secret-telemeter-client\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.032989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.032935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-federate-client-tls\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.033171 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.033013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-metrics-client-ca\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.133740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-serving-certs-ca-bundle\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.133740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-telemeter-client-tls\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.133740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5fn\" (UniqueName: \"kubernetes.io/projected/94edbbbf-ee22-4f2e-9444-634d7faec842-kube-api-access-8s5fn\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.133740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-secret-telemeter-client\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.134053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-federate-client-tls\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.134053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-metrics-client-ca\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.134053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-telemeter-trusted-ca-bundle\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.134053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.133844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.134504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.134444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-serving-certs-ca-bundle\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.134733 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.134706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-metrics-client-ca\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.135106 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.135081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94edbbbf-ee22-4f2e-9444-634d7faec842-telemeter-trusted-ca-bundle\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.136577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.136549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-secret-telemeter-client\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.136658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.136588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.136699 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.136679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-federate-client-tls\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.136734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.136706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/94edbbbf-ee22-4f2e-9444-634d7faec842-telemeter-client-tls\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.141061 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.141045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5fn\" (UniqueName: \"kubernetes.io/projected/94edbbbf-ee22-4f2e-9444-634d7faec842-kube-api-access-8s5fn\") pod \"telemeter-client-589dd97556-tc7hb\" (UID: \"94edbbbf-ee22-4f2e-9444-634d7faec842\") " pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.275058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.275026    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb"
Sep 12 00:22:26.553866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.553836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-589dd97556-tc7hb"]
Sep 12 00:22:26.554444 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:22:26.554419    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94edbbbf_ee22_4f2e_9444_634d7faec842.slice/crio-59335f5650d471aff786a50072734f08e2a7b10be2e0b0a90c467b5e8f02992f WatchSource:0}: Error finding container 59335f5650d471aff786a50072734f08e2a7b10be2e0b0a90c467b5e8f02992f: Status 404 returned error can't find the container with id 59335f5650d471aff786a50072734f08e2a7b10be2e0b0a90c467b5e8f02992f
Sep 12 00:22:26.556595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.556579    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:22:26.835546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:26.835508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb" event={"ID":"94edbbbf-ee22-4f2e-9444-634d7faec842","Type":"ContainerStarted","Data":"59335f5650d471aff786a50072734f08e2a7b10be2e0b0a90c467b5e8f02992f"}
Sep 12 00:22:29.849325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:29.849284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb" event={"ID":"94edbbbf-ee22-4f2e-9444-634d7faec842","Type":"ContainerStarted","Data":"3041fabbc0f825f7606572195d3f4ff603b6f490638b7595b10c179f499aee66"}
Sep 12 00:22:29.849760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:29.849349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb" event={"ID":"94edbbbf-ee22-4f2e-9444-634d7faec842","Type":"ContainerStarted","Data":"8b48079c687b8ab89a0559d974d743e1166a8ef705ccbcfd2ec39d77861fb952"}
Sep 12 00:22:29.849760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:29.849371    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb" event={"ID":"94edbbbf-ee22-4f2e-9444-634d7faec842","Type":"ContainerStarted","Data":"3c50fcbda1578efe154faa41e56cdf6507660a5021283e16ef1ee03550afc81c"}
Sep 12 00:22:29.870713 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:29.870666    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-589dd97556-tc7hb" podStartSLOduration=2.622746919 podStartE2EDuration="4.870649676s" totalImagesPullingTime="2.247902757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:22:29.869778715 +0000 UTC m=+352.807085234" watchObservedRunningTime="2026-09-12 00:22:29.870649676 +0000 UTC m=+352.807956196"
Sep 12 00:22:30.002881 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:30.002672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:30.057225 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:30.057190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:30.188335 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:30.188249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:30.259499 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:30.259446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:30.752466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:30.752434    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:31.267554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:31.267520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:39.902164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:39.902114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:39.968051 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:39.968022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:40.029862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:40.029838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:40.605109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:40.605072    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5757bd6795-hhp5k"]
Sep 12 00:22:40.707378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:40.707346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:40.758835 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:40.758805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:40.911570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:40.911496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:44.600771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:44.600729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:22:44.661759 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:22:44.661719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:05.624654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.624592    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5757bd6795-hhp5k" podUID="f98b40aa-e821-4ac1-9231-4be9334e370f" containerName="console" containerID="cri-o://aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911" gracePeriod=15
Sep 12 00:23:05.869809 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.869784    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:23:05.955072 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.954992    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-serving-cert\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955072 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-trusted-ca-bundle\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955090    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-console-config\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955106    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-service-ca\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955162    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhhm6\" (UniqueName: \"kubernetes.io/projected/f98b40aa-e821-4ac1-9231-4be9334e370f-kube-api-access-fhhm6\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955198    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-oauth-config\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955220    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-oauth-serving-cert\") pod \"f98b40aa-e821-4ac1-9231-4be9334e370f\" (UID: \"f98b40aa-e821-4ac1-9231-4be9334e370f\") "
Sep 12 00:23:05.955675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955545    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-service-ca" (OuterVolumeSpecName: "service-ca") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:23:05.955675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955562    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-console-config" (OuterVolumeSpecName: "console-config") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:23:05.955675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:23:05.955904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.955736    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:23:05.957423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.957396    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:23:05.957525 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.957418    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b40aa-e821-4ac1-9231-4be9334e370f-kube-api-access-fhhm6" (OuterVolumeSpecName: "kube-api-access-fhhm6") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "kube-api-access-fhhm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:23:05.957732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.957706    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f98b40aa-e821-4ac1-9231-4be9334e370f" (UID: "f98b40aa-e821-4ac1-9231-4be9334e370f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:23:05.994702 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.994670    2562 generic.go:356] "Generic (PLEG): container finished" podID="f98b40aa-e821-4ac1-9231-4be9334e370f" containerID="aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911" exitCode=0
Sep 12 00:23:05.994830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.994718    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5757bd6795-hhp5k"
Sep 12 00:23:05.994830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.994758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5757bd6795-hhp5k" event={"ID":"f98b40aa-e821-4ac1-9231-4be9334e370f","Type":"ContainerDied","Data":"aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911"}
Sep 12 00:23:05.994830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.994807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5757bd6795-hhp5k" event={"ID":"f98b40aa-e821-4ac1-9231-4be9334e370f","Type":"ContainerDied","Data":"9b760bc5621a8700509bbf5f47904fb15932cd44205e1dc5f986e082c1d41498"}
Sep 12 00:23:05.994830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:05.994826    2562 scope.go:117] "RemoveContainer" containerID="aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911"
Sep 12 00:23:06.002871 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.002856    2562 scope.go:117] "RemoveContainer" containerID="aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911"
Sep 12 00:23:06.003116 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:23:06.003098    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911\": container with ID starting with aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911 not found: ID does not exist" containerID="aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911"
Sep 12 00:23:06.003158 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.003122    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911"} err="failed to get container status \"aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911\": rpc error: code = NotFound desc = could not find container \"aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911\": container with ID starting with aefe8cb075f6a5535ef0e6ff8656247f2b5c324875d81df12f6888c64f93c911 not found: ID does not exist"
Sep 12 00:23:06.016217 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.016195    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5757bd6795-hhp5k"]
Sep 12 00:23:06.019546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.019528    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5757bd6795-hhp5k"]
Sep 12 00:23:06.056208 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056185    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhhm6\" (UniqueName: \"kubernetes.io/projected/f98b40aa-e821-4ac1-9231-4be9334e370f-kube-api-access-fhhm6\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:06.056208 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056207    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-oauth-config\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:06.056327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056218    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-oauth-serving-cert\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:06.056327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056227    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f98b40aa-e821-4ac1-9231-4be9334e370f-console-serving-cert\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:06.056327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056235    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-trusted-ca-bundle\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:06.056327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056244    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-console-config\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:06.056327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:06.056252    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f98b40aa-e821-4ac1-9231-4be9334e370f-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:23:07.568659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:07.568624    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98b40aa-e821-4ac1-9231-4be9334e370f" path="/var/lib/kubelet/pods/f98b40aa-e821-4ac1-9231-4be9334e370f/volumes"
Sep 12 00:23:30.768084 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:30.768052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:30.827930 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:30.827898    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:36.115942 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:36.115911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:36.166561 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:36.166531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:38.618355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.618320    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cb98bb9d-7jzhb"]
Sep 12 00:23:38.619389 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.619355    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98b40aa-e821-4ac1-9231-4be9334e370f" containerName="console"
Sep 12 00:23:38.619389 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.619386    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98b40aa-e821-4ac1-9231-4be9334e370f" containerName="console"
Sep 12 00:23:38.619570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.619505    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98b40aa-e821-4ac1-9231-4be9334e370f" containerName="console"
Sep 12 00:23:38.623830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.623809    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.626461 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.626443    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 12 00:23:38.626593 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.626460    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bsxwp\""
Sep 12 00:23:38.626728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.626714    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 12 00:23:38.628019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.628000    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 12 00:23:38.628289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.627999    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 12 00:23:38.628370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.628024    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 12 00:23:38.628428 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.628065    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 12 00:23:38.628663 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.628648    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 12 00:23:38.632645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.632622    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 12 00:23:38.632897 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.632876    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cb98bb9d-7jzhb"]
Sep 12 00:23:38.665380 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.665353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:38.698350 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-service-ca\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.698446 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-oauth-config\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.698446 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-serving-cert\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.698571 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zzh\" (UniqueName: \"kubernetes.io/projected/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-kube-api-access-g2zzh\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.698571 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-trusted-ca-bundle\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.698674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-oauth-serving-cert\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.698674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.698619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-config\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.717811 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.717788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:38.799116 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-serving-cert\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.799116 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799118    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zzh\" (UniqueName: \"kubernetes.io/projected/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-kube-api-access-g2zzh\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.799303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-trusted-ca-bundle\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.799303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-oauth-serving-cert\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.799303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-config\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.799303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-service-ca\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.799303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.799288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-oauth-config\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.800084 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.800055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-oauth-serving-cert\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.800189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.800086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-config\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.800189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.800110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-service-ca\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.800189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.800174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-trusted-ca-bundle\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.801645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.801620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-serving-cert\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.801645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.801632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-oauth-config\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.812567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.812550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zzh\" (UniqueName: \"kubernetes.io/projected/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-kube-api-access-g2zzh\") pod \"console-54cb98bb9d-7jzhb\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") " pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:38.935086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:38.935011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:39.204552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:39.204460    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cb98bb9d-7jzhb"]
Sep 12 00:23:39.208293 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:23:39.208265    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9f7ef2_e3cd_47a9_979f_21c13a3871df.slice/crio-848b809989c40fce0a7cccceb04805ae5ee340b9ce5d3443b3720cd2fee7b0db WatchSource:0}: Error finding container 848b809989c40fce0a7cccceb04805ae5ee340b9ce5d3443b3720cd2fee7b0db: Status 404 returned error can't find the container with id 848b809989c40fce0a7cccceb04805ae5ee340b9ce5d3443b3720cd2fee7b0db
Sep 12 00:23:40.126685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:40.126653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cb98bb9d-7jzhb" event={"ID":"3e9f7ef2-e3cd-47a9-979f-21c13a3871df","Type":"ContainerStarted","Data":"9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c"}
Sep 12 00:23:40.127143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:40.126699    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cb98bb9d-7jzhb" event={"ID":"3e9f7ef2-e3cd-47a9-979f-21c13a3871df","Type":"ContainerStarted","Data":"848b809989c40fce0a7cccceb04805ae5ee340b9ce5d3443b3720cd2fee7b0db"}
Sep 12 00:23:40.144027 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:40.143981    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54cb98bb9d-7jzhb" podStartSLOduration=2.143969053 podStartE2EDuration="2.143969053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:23:40.142279563 +0000 UTC m=+423.079586083" watchObservedRunningTime="2026-09-12 00:23:40.143969053 +0000 UTC m=+423.081275573"
Sep 12 00:23:41.494906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:41.494872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:41.555674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:41.555642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:47.983119 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:47.983088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:48.032687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:48.032644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:48.935361 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:48.935324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:48.935537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:48.935381    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:48.940160 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:48.940134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:49.163979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:49.163954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:23:49.314210 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:49.314186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:49.374936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:49.374906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:49.433417 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:49.433384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:54.912905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:54.912874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:23:55.012290 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:23:55.012260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:24:50.548046 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:24:50.548013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:24:50.642816 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:24:50.642785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:24:57.689390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:24:57.689354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:12.100372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:12.100333    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:12.867890 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:12.867838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:15.704166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:15.704133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:15.755466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:15.755439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:17.567170 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:17.567140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:17.695729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:17.695674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:17.985031 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:17.984953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:18.081562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:18.081529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:19.978456 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:19.978426    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:20.370688 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.370656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:20.583165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.583134    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"]
Sep 12 00:25:20.586224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.586206    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.588411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.588389    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 12 00:25:20.588558 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.588502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 12 00:25:20.589437 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.589421    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nkdpq\""
Sep 12 00:25:20.594637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.594612    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"]
Sep 12 00:25:20.692775 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.692708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77284f25-aca8-4c3a-b17d-87068233bb84-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nmnpr\" (UID: \"77284f25-aca8-4c3a-b17d-87068233bb84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.692961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.692944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh47q\" (UniqueName: \"kubernetes.io/projected/77284f25-aca8-4c3a-b17d-87068233bb84-kube-api-access-fh47q\") pod \"cert-manager-cainjector-559cdd8c87-nmnpr\" (UID: \"77284f25-aca8-4c3a-b17d-87068233bb84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.794184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.794154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh47q\" (UniqueName: \"kubernetes.io/projected/77284f25-aca8-4c3a-b17d-87068233bb84-kube-api-access-fh47q\") pod \"cert-manager-cainjector-559cdd8c87-nmnpr\" (UID: \"77284f25-aca8-4c3a-b17d-87068233bb84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.794328 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.794214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77284f25-aca8-4c3a-b17d-87068233bb84-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nmnpr\" (UID: \"77284f25-aca8-4c3a-b17d-87068233bb84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.802223 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.802201    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77284f25-aca8-4c3a-b17d-87068233bb84-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nmnpr\" (UID: \"77284f25-aca8-4c3a-b17d-87068233bb84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.802538 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.802516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh47q\" (UniqueName: \"kubernetes.io/projected/77284f25-aca8-4c3a-b17d-87068233bb84-kube-api-access-fh47q\") pod \"cert-manager-cainjector-559cdd8c87-nmnpr\" (UID: \"77284f25-aca8-4c3a-b17d-87068233bb84\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:20.894797 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:20.894772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"
Sep 12 00:25:21.164770 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:21.164748    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr"]
Sep 12 00:25:21.166973 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:25:21.166945    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77284f25_aca8_4c3a_b17d_87068233bb84.slice/crio-1dce9f29406561766e450333ab7fe426f34aa2c1f81c62873585db1684105f91 WatchSource:0}: Error finding container 1dce9f29406561766e450333ab7fe426f34aa2c1f81c62873585db1684105f91: Status 404 returned error can't find the container with id 1dce9f29406561766e450333ab7fe426f34aa2c1f81c62873585db1684105f91
Sep 12 00:25:21.533326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:21.533289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr" event={"ID":"77284f25-aca8-4c3a-b17d-87068233bb84","Type":"ContainerStarted","Data":"1dce9f29406561766e450333ab7fe426f34aa2c1f81c62873585db1684105f91"}
Sep 12 00:25:22.500730 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:22.500696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:22.601424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:22.601386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:24.549455 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:24.549394    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr" event={"ID":"77284f25-aca8-4c3a-b17d-87068233bb84","Type":"ContainerStarted","Data":"c785cbfb60971fd9b3b2b9c6f04289f1f1a63cf2e612fb796d20d67065b1a39c"}
Sep 12 00:25:24.565972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:24.565924    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nmnpr" podStartSLOduration=1.327282339 podStartE2EDuration="4.56590965s" totalImagesPullingTime="3.238627311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:25:24.564017095 +0000 UTC m=+527.501323609" watchObservedRunningTime="2026-09-12 00:25:24.56590965 +0000 UTC m=+527.503216170"
Sep 12 00:25:28.588908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:28.588880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:28.979327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:28.979248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:29.042433 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:29.042375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:30.172233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:30.172152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:30.268885 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:30.268852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:32.007333 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.007277    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"]
Sep 12 00:25:32.010131 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.010116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.012511 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.012483    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-bnjp5\""
Sep 12 00:25:32.012624 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.012516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 12 00:25:32.013512 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.013492    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:25:32.017944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.017925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"]
Sep 12 00:25:32.187030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.186996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6684d846-3d3a-4cee-baf5-c597a1f2e9b0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-znjlz\" (UID: \"6684d846-3d3a-4cee-baf5-c597a1f2e9b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.187228 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.187056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zh7\" (UniqueName: \"kubernetes.io/projected/6684d846-3d3a-4cee-baf5-c597a1f2e9b0-kube-api-access-m6zh7\") pod \"openshift-lws-operator-fd8ccff4c-znjlz\" (UID: \"6684d846-3d3a-4cee-baf5-c597a1f2e9b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.288182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.288098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6684d846-3d3a-4cee-baf5-c597a1f2e9b0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-znjlz\" (UID: \"6684d846-3d3a-4cee-baf5-c597a1f2e9b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.288182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.288151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zh7\" (UniqueName: \"kubernetes.io/projected/6684d846-3d3a-4cee-baf5-c597a1f2e9b0-kube-api-access-m6zh7\") pod \"openshift-lws-operator-fd8ccff4c-znjlz\" (UID: \"6684d846-3d3a-4cee-baf5-c597a1f2e9b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.288516 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.288496    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6684d846-3d3a-4cee-baf5-c597a1f2e9b0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-znjlz\" (UID: \"6684d846-3d3a-4cee-baf5-c597a1f2e9b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.295739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.295713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zh7\" (UniqueName: \"kubernetes.io/projected/6684d846-3d3a-4cee-baf5-c597a1f2e9b0-kube-api-access-m6zh7\") pod \"openshift-lws-operator-fd8ccff4c-znjlz\" (UID: \"6684d846-3d3a-4cee-baf5-c597a1f2e9b0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.319562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.319524    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"
Sep 12 00:25:32.586504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:32.586458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz"]
Sep 12 00:25:32.589174 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:25:32.589137    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6684d846_3d3a_4cee_baf5_c597a1f2e9b0.slice/crio-c2b91a8c34bd4636501c0688aa2dc3172973057ba3253c654fc5766f5236ee37 WatchSource:0}: Error finding container c2b91a8c34bd4636501c0688aa2dc3172973057ba3253c654fc5766f5236ee37: Status 404 returned error can't find the container with id c2b91a8c34bd4636501c0688aa2dc3172973057ba3253c654fc5766f5236ee37
Sep 12 00:25:33.587285 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:33.587229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz" event={"ID":"6684d846-3d3a-4cee-baf5-c597a1f2e9b0","Type":"ContainerStarted","Data":"c2b91a8c34bd4636501c0688aa2dc3172973057ba3253c654fc5766f5236ee37"}
Sep 12 00:25:35.596660 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:35.596619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz" event={"ID":"6684d846-3d3a-4cee-baf5-c597a1f2e9b0","Type":"ContainerStarted","Data":"3b33e51daeeefbc79598910f3723ebec21be2e32b97b4880f6f4929cea68c160"}
Sep 12 00:25:35.612684 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:35.612633    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-znjlz" podStartSLOduration=2.396797605 podStartE2EDuration="4.612618374s" totalImagesPullingTime="2.215820769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:25:35.61139675 +0000 UTC m=+538.548703270" watchObservedRunningTime="2026-09-12 00:25:35.612618374 +0000 UTC m=+538.549924894"
Sep 12 00:25:36.334540 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:36.334506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:36.439776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:36.439743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:38.061025 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:38.060965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:53.008286 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:53.008245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:54.333291 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:54.333230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:54.451109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:54.451073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.151282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.151244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.387163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.387130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.490917 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.490843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.512846 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.512812    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"]
Sep 12 00:25:55.519200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.519176    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.521704 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.521683    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 00:25:55.521825 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.521811    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 00:25:55.521896 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.521812    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 00:25:55.521896 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.521860    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 00:25:55.522000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.521985    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mb7xc\""
Sep 12 00:25:55.528611 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.528591    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"]
Sep 12 00:25:55.560925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.560899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4e6a6c5-8374-4047-8472-7372b0aed510-tmp\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.561063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.560939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4e6a6c5-8374-4047-8472-7372b0aed510-webhook-cert\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.561063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.560960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4e6a6c5-8374-4047-8472-7372b0aed510-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.561063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.560976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4e6a6c5-8374-4047-8472-7372b0aed510-manifests\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.561205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.561102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4jr\" (UniqueName: \"kubernetes.io/projected/c4e6a6c5-8374-4047-8472-7372b0aed510-kube-api-access-zz4jr\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.625181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.625149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.661923 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.661896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4jr\" (UniqueName: \"kubernetes.io/projected/c4e6a6c5-8374-4047-8472-7372b0aed510-kube-api-access-zz4jr\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.662034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.661965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4e6a6c5-8374-4047-8472-7372b0aed510-tmp\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.662096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.662030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4e6a6c5-8374-4047-8472-7372b0aed510-webhook-cert\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.662096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.662064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4e6a6c5-8374-4047-8472-7372b0aed510-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.662211 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.662093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4e6a6c5-8374-4047-8472-7372b0aed510-manifests\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.662525 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.662505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4e6a6c5-8374-4047-8472-7372b0aed510-manifests\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.664185 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.664160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4e6a6c5-8374-4047-8472-7372b0aed510-tmp\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.664449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.664430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4e6a6c5-8374-4047-8472-7372b0aed510-webhook-cert\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.664449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.664442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4e6a6c5-8374-4047-8472-7372b0aed510-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.678606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.678586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4jr\" (UniqueName: \"kubernetes.io/projected/c4e6a6c5-8374-4047-8472-7372b0aed510-kube-api-access-zz4jr\") pod \"opendatahub-operator-controller-manager-8446f459d6-p9cwf\" (UID: \"c4e6a6c5-8374-4047-8472-7372b0aed510\") " pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.725859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.725806    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.829109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.829084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:55.830010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.829993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:25:55.927085 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:55.927053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:56.028362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:56.028332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:25:56.103553 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:56.103525    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"]
Sep 12 00:25:56.103885 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:25:56.103861    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e6a6c5_8374_4047_8472_7372b0aed510.slice/crio-2328abdc41dafa8c3d95ea77cf18eef603eb46588abe7e6911a4d25f32efbb70 WatchSource:0}: Error finding container 2328abdc41dafa8c3d95ea77cf18eef603eb46588abe7e6911a4d25f32efbb70: Status 404 returned error can't find the container with id 2328abdc41dafa8c3d95ea77cf18eef603eb46588abe7e6911a4d25f32efbb70
Sep 12 00:25:56.685296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:56.685256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf" event={"ID":"c4e6a6c5-8374-4047-8472-7372b0aed510","Type":"ContainerStarted","Data":"2328abdc41dafa8c3d95ea77cf18eef603eb46588abe7e6911a4d25f32efbb70"}
Sep 12 00:25:59.699795 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:25:59.699759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf" event={"ID":"c4e6a6c5-8374-4047-8472-7372b0aed510","Type":"ContainerStarted","Data":"f4137bd1f4a981ab45a908bbc6c631be8168c7c27a4bdbd9417ae6fb67cccff3"}
Sep 12 00:26:00.705707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:00.705623    2562 generic.go:356] "Generic (PLEG): container finished" podID="c4e6a6c5-8374-4047-8472-7372b0aed510" containerID="f4137bd1f4a981ab45a908bbc6c631be8168c7c27a4bdbd9417ae6fb67cccff3" exitCode=0
Sep 12 00:26:00.705707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:00.705697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf" event={"ID":"c4e6a6c5-8374-4047-8472-7372b0aed510","Type":"ContainerDied","Data":"f4137bd1f4a981ab45a908bbc6c631be8168c7c27a4bdbd9417ae6fb67cccff3"}
Sep 12 00:26:01.712489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:01.712449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf" event={"ID":"c4e6a6c5-8374-4047-8472-7372b0aed510","Type":"ContainerStarted","Data":"87deaa80830223bd69b17e33db2a1799cd25589cf268cbb372dd04f000755527"}
Sep 12 00:26:01.712878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:01.712593    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:26:01.735408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:01.735360    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf" podStartSLOduration=2.207626558 podStartE2EDuration="6.735346624s" totalImagesPullingTime="3.517442426s" totalInitContainerRuntime="1.01027764s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:25:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:26:01.734005298 +0000 UTC m=+564.671311819" watchObservedRunningTime="2026-09-12 00:26:01.735346624 +0000 UTC m=+564.672653182"
Sep 12 00:26:01.876538 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:01.876499    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:01.977182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:01.977092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:07.603800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:07.603747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:07.701564 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:07.701532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:09.103146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:09.103109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:09.210893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:09.210863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:09.303493 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:09.303447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:09.836212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:09.836168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:12.722137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:12.722105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8446f459d6-p9cwf"
Sep 12 00:26:14.971126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:14.971073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:15.090567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:15.090531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:15.313664 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:15.313581    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:15.854101 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:15.854059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:16.511946 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.511912    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"]
Sep 12 00:26:16.516705 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.516680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.517125 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.517091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:16.518940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.518925    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 12 00:26:16.519172 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.519155    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 12 00:26:16.519262 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.519162    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 12 00:26:16.519332 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.519293    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 00:26:16.519332 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.519307    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qt589\""
Sep 12 00:26:16.519440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.519342    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 12 00:26:16.519699 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.519684    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 00:26:16.527289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.527265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"]
Sep 12 00:26:16.579577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.579539    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:16.638973 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.638944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.639081 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.638980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.639081 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.639012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9jz\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-kube-api-access-4l9jz\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.639081 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.639074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.639244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.639137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.639244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.639168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.639244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.639207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740335 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740389 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9jz\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-kube-api-access-4l9jz\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740445 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740445 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740560 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.740816 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.740779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.742642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.742611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.742783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.742738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.742783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.742773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.742892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.742808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.748620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.748585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.748803 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.748786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9jz\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-kube-api-access-4l9jz\") pod \"istiod-openshift-gateway-7b7995d5cd-q8zww\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:16.825495 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:16.825457    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:17.096364 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:17.096341    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"]
Sep 12 00:26:17.098243 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:26:17.098212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d1b43f_bc82_41b3_9c29_5a7c8285d465.slice/crio-283429fc8b5c31cbe377d0c701cc7aef7a074363fc1070bf1eeaf6ea797dde29 WatchSource:0}: Error finding container 283429fc8b5c31cbe377d0c701cc7aef7a074363fc1070bf1eeaf6ea797dde29: Status 404 returned error can't find the container with id 283429fc8b5c31cbe377d0c701cc7aef7a074363fc1070bf1eeaf6ea797dde29
Sep 12 00:26:17.367756 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:17.367675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:17.791513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:17.791398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" event={"ID":"e3d1b43f-bc82-41b3-9c29-5a7c8285d465","Type":"ContainerStarted","Data":"283429fc8b5c31cbe377d0c701cc7aef7a074363fc1070bf1eeaf6ea797dde29"}
Sep 12 00:26:19.466657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.466514    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cb98bb9d-7jzhb"]
Sep 12 00:26:19.569288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.569229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:19.660129 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.660090    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:26:19.660239 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.660155    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:26:19.678038 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.678005    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:19.778814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.778741    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:19.802757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.802722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" event={"ID":"e3d1b43f-bc82-41b3-9c29-5a7c8285d465","Type":"ContainerStarted","Data":"dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc"}
Sep 12 00:26:19.802907 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.802839    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:19.820522 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:19.820435    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" podStartSLOduration=1.260862429 podStartE2EDuration="3.820419683s" totalImagesPullingTime="2.559557254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:26:19.819642502 +0000 UTC m=+582.756949022" watchObservedRunningTime="2026-09-12 00:26:19.820419683 +0000 UTC m=+582.757726203"
Sep 12 00:26:20.382597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:20.382560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:20.446950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:20.446915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:20.809112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:20.809011    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-q8zww container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 00:26:20.809112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:20.809075    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" podUID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:26:20.936950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:20.936907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:21.036861 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:21.036828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:21.136734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:21.136699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:23.808977 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:23.808945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:26.058264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:26.058224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:26.161163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:26.161131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:26.259184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:26.259151    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:26.359974 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:26.359938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:26.552765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:26.552737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:26.758185 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:26.758108    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:33.043793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.043759    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"]
Sep 12 00:26:33.067172 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.067130    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"]
Sep 12 00:26:33.067300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.067244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.070393 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.070368    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 12 00:26:33.071298 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.071269    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 12 00:26:33.071419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.071311    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 12 00:26:33.071419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.071284    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-28vf9\""
Sep 12 00:26:33.086454 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.086429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8bx\" (UniqueName: \"kubernetes.io/projected/0ef40f12-355a-4fc0-9158-afba7b521387-kube-api-access-nq8bx\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.086555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.086500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef40f12-355a-4fc0-9158-afba7b521387-cert\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.086555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.086519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ef40f12-355a-4fc0-9158-afba7b521387-metrics-cert\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.086629 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.086558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ef40f12-355a-4fc0-9158-afba7b521387-manager-config\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.187761 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.187736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ef40f12-355a-4fc0-9158-afba7b521387-manager-config\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.187906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.187782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8bx\" (UniqueName: \"kubernetes.io/projected/0ef40f12-355a-4fc0-9158-afba7b521387-kube-api-access-nq8bx\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.187906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.187854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef40f12-355a-4fc0-9158-afba7b521387-cert\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.187906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.187880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ef40f12-355a-4fc0-9158-afba7b521387-metrics-cert\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.188378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.188354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ef40f12-355a-4fc0-9158-afba7b521387-manager-config\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.190346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.190327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ef40f12-355a-4fc0-9158-afba7b521387-metrics-cert\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.190424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.190326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef40f12-355a-4fc0-9158-afba7b521387-cert\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.195612 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.195592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8bx\" (UniqueName: \"kubernetes.io/projected/0ef40f12-355a-4fc0-9158-afba7b521387-kube-api-access-nq8bx\") pod \"lws-controller-manager-7b5cf77b9-n9jgm\" (UID: \"0ef40f12-355a-4fc0-9158-afba7b521387\") " pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.377314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.377273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:33.649516 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.649488    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"]
Sep 12 00:26:33.652682 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:26:33.652652    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef40f12_355a_4fc0_9158_afba7b521387.slice/crio-f474a6bd5314635bccc9a2e2205c1b942b90428578266b090266a3506b4daeb9 WatchSource:0}: Error finding container f474a6bd5314635bccc9a2e2205c1b942b90428578266b090266a3506b4daeb9: Status 404 returned error can't find the container with id f474a6bd5314635bccc9a2e2205c1b942b90428578266b090266a3506b4daeb9
Sep 12 00:26:33.869753 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:33.869719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm" event={"ID":"0ef40f12-355a-4fc0-9158-afba7b521387","Type":"ContainerStarted","Data":"f474a6bd5314635bccc9a2e2205c1b942b90428578266b090266a3506b4daeb9"}
Sep 12 00:26:34.935770 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:34.935733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:35.065745 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:35.065694    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:35.395880 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:35.395845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:35.726889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:35.726851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:35.880875 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:35.880839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm" event={"ID":"0ef40f12-355a-4fc0-9158-afba7b521387","Type":"ContainerStarted","Data":"311f2567985576090483a321db66a8f0a2d2d4f2e9fef20b06afb7fbddb238bd"}
Sep 12 00:26:35.881049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:35.880931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:36.227133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:36.227101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:36.325765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:36.325729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:37.460063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.460033    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:26:37.460915 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.460895    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:26:37.464402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.464332    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm" podStartSLOduration=2.8547199020000003 podStartE2EDuration="4.464313134s" totalImagesPullingTime="1.609593232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:26:35.910896177 +0000 UTC m=+598.848202697" watchObservedRunningTime="2026-09-12 00:26:37.464313134 +0000 UTC m=+600.401619655"
Sep 12 00:26:37.464562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.464540    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"]
Sep 12 00:26:37.464848 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.464819    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" podUID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" containerName="discovery" containerID="cri-o://dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc" gracePeriod=30
Sep 12 00:26:37.712975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.712918    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:37.730059 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730034    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l9jz\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-kube-api-access-4l9jz\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730070    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-cacerts\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730086    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-dns-cert\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-ca-configmap\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730293 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730170    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-kubeconfig\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730293 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730184    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-token\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730293 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730249    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-local-certs\") pod \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\" (UID: \"e3d1b43f-bc82-41b3-9c29-5a7c8285d465\") "
Sep 12 00:26:37.730597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.730554    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:26:37.732405 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.732376    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:26:37.732555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.732413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-kube-api-access-4l9jz" (OuterVolumeSpecName: "kube-api-access-4l9jz") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "kube-api-access-4l9jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:26:37.732622 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.732573    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-cacerts" (OuterVolumeSpecName: "cacerts") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:26:37.732734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.732707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:26:37.732788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.732733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:26:37.732847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.732798    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-local-certs" (OuterVolumeSpecName: "local-certs") pod "e3d1b43f-bc82-41b3-9c29-5a7c8285d465" (UID: "e3d1b43f-bc82-41b3-9c29-5a7c8285d465"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:26:37.831050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831017    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-ca-configmap\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.831050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831043    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-kubeconfig\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.831227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831056    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-token\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.831227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831069    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-local-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.831227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831079    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9jz\" (UniqueName: \"kubernetes.io/projected/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-kube-api-access-4l9jz\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.831227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831090    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-cacerts\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.831227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.831102    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3d1b43f-bc82-41b3-9c29-5a7c8285d465-istio-csr-dns-cert\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:37.890358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.890322    2562 generic.go:356] "Generic (PLEG): container finished" podID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" containerID="dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc" exitCode=0
Sep 12 00:26:37.890494 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.890355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" event={"ID":"e3d1b43f-bc82-41b3-9c29-5a7c8285d465","Type":"ContainerDied","Data":"dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc"}
Sep 12 00:26:37.890494 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.890411    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww" event={"ID":"e3d1b43f-bc82-41b3-9c29-5a7c8285d465","Type":"ContainerDied","Data":"283429fc8b5c31cbe377d0c701cc7aef7a074363fc1070bf1eeaf6ea797dde29"}
Sep 12 00:26:37.890494 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.890365    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"
Sep 12 00:26:37.890494 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.890437    2562 scope.go:117] "RemoveContainer" containerID="dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc"
Sep 12 00:26:37.899563 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.899544    2562 scope.go:117] "RemoveContainer" containerID="dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc"
Sep 12 00:26:37.899810 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:26:37.899791    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc\": container with ID starting with dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc not found: ID does not exist" containerID="dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc"
Sep 12 00:26:37.899879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.899822    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc"} err="failed to get container status \"dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc\": rpc error: code = NotFound desc = could not find container \"dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc\": container with ID starting with dcca1359a8e6573e2dba4c582874d2cc9a786de5e5a65cfac6db6c83b0d190dc not found: ID does not exist"
Sep 12 00:26:37.912061 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.912036    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"]
Sep 12 00:26:37.916158 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:37.916137    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q8zww"]
Sep 12 00:26:39.569933 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:39.569897    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" path="/var/lib/kubelet/pods/e3d1b43f-bc82-41b3-9c29-5a7c8285d465/volumes"
Sep 12 00:26:44.491652 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.491603    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54cb98bb9d-7jzhb" podUID="3e9f7ef2-e3cd-47a9-979f-21c13a3871df" containerName="console" containerID="cri-o://9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c" gracePeriod=15
Sep 12 00:26:44.735289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.735260    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:26:44.783794 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783726    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-serving-cert\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.783794 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-trusted-ca-bundle\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.783941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783818    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-oauth-serving-cert\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.783941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783853    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-service-ca\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.783941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783869    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-config\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.783941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783902    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-oauth-config\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.783941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.783929    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2zzh\" (UniqueName: \"kubernetes.io/projected/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-kube-api-access-g2zzh\") pod \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\" (UID: \"3e9f7ef2-e3cd-47a9-979f-21c13a3871df\") "
Sep 12 00:26:44.784291 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.784259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:26:44.784291 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.784284    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:26:44.784440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.784279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-service-ca" (OuterVolumeSpecName: "service-ca") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:26:44.784440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.784302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-config" (OuterVolumeSpecName: "console-config") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:26:44.785967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.785936    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:26:44.785967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.785953    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:26:44.786089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.786011    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-kube-api-access-g2zzh" (OuterVolumeSpecName: "kube-api-access-g2zzh") pod "3e9f7ef2-e3cd-47a9-979f-21c13a3871df" (UID: "3e9f7ef2-e3cd-47a9-979f-21c13a3871df"). InnerVolumeSpecName "kube-api-access-g2zzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:26:44.885266 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885231    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-oauth-serving-cert\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.885266 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885260    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.885266 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885271    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-config\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.885503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885279    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-oauth-config\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.885503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885288    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2zzh\" (UniqueName: \"kubernetes.io/projected/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-kube-api-access-g2zzh\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.885503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885297    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-console-serving-cert\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.885503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.885306    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e9f7ef2-e3cd-47a9-979f-21c13a3871df-trusted-ca-bundle\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:26:44.922705 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.922669    2562 generic.go:356] "Generic (PLEG): container finished" podID="3e9f7ef2-e3cd-47a9-979f-21c13a3871df" containerID="9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c" exitCode=0
Sep 12 00:26:44.922855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.922710    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cb98bb9d-7jzhb"
Sep 12 00:26:44.922855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.922755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cb98bb9d-7jzhb" event={"ID":"3e9f7ef2-e3cd-47a9-979f-21c13a3871df","Type":"ContainerDied","Data":"9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c"}
Sep 12 00:26:44.922855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.922826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cb98bb9d-7jzhb" event={"ID":"3e9f7ef2-e3cd-47a9-979f-21c13a3871df","Type":"ContainerDied","Data":"848b809989c40fce0a7cccceb04805ae5ee340b9ce5d3443b3720cd2fee7b0db"}
Sep 12 00:26:44.922855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.922853    2562 scope.go:117] "RemoveContainer" containerID="9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c"
Sep 12 00:26:44.931998 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.931982    2562 scope.go:117] "RemoveContainer" containerID="9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c"
Sep 12 00:26:44.932284 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:26:44.932263    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c\": container with ID starting with 9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c not found: ID does not exist" containerID="9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c"
Sep 12 00:26:44.932351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.932294    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c"} err="failed to get container status \"9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c\": rpc error: code = NotFound desc = could not find container \"9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c\": container with ID starting with 9148a4cfe4e97e5eb836fcfce62ce0b484eeae0f036c80621725f5e081b8bc3c not found: ID does not exist"
Sep 12 00:26:44.943547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.943522    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cb98bb9d-7jzhb"]
Sep 12 00:26:44.946506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:44.946486    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54cb98bb9d-7jzhb"]
Sep 12 00:26:45.569932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:45.569892    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9f7ef2-e3cd-47a9-979f-21c13a3871df" path="/var/lib/kubelet/pods/3e9f7ef2-e3cd-47a9-979f-21c13a3871df/volumes"
Sep 12 00:26:46.887273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:46.887237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b5cf77b9-n9jgm"
Sep 12 00:26:57.024871 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:57.024837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:26:58.260833 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:26:58.260797    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:00.305484 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:00.305394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:00.406338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:00.406304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:10.294865 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:10.294829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:27.185501 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:27.185457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:27.286715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:27.286676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:40.511719 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:40.511685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:42.532821 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.532788    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"]
Sep 12 00:27:42.533202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.533107    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" containerName="discovery"
Sep 12 00:27:42.533202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.533118    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" containerName="discovery"
Sep 12 00:27:42.533202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.533139    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9f7ef2-e3cd-47a9-979f-21c13a3871df" containerName="console"
Sep 12 00:27:42.533202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.533144    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9f7ef2-e3cd-47a9-979f-21c13a3871df" containerName="console"
Sep 12 00:27:42.533202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.533194    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d1b43f-bc82-41b3-9c29-5a7c8285d465" containerName="discovery"
Sep 12 00:27:42.533202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.533202    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9f7ef2-e3cd-47a9-979f-21c13a3871df" containerName="console"
Sep 12 00:27:42.535663 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.535647    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:27:42.538613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.538589    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-k85tr\""
Sep 12 00:27:42.539765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.539742    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 00:27:42.539765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.539760    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 00:27:42.546376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.546349    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"]
Sep 12 00:27:42.566999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.566970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kfl\" (UniqueName: \"kubernetes.io/projected/31534654-7557-4a65-b042-00d57f87e2dd-kube-api-access-d9kfl\") pod \"limitador-operator-controller-manager-74495c69f8-72m4k\" (UID: \"31534654-7557-4a65-b042-00d57f87e2dd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:27:42.667442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.667399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kfl\" (UniqueName: \"kubernetes.io/projected/31534654-7557-4a65-b042-00d57f87e2dd-kube-api-access-d9kfl\") pod \"limitador-operator-controller-manager-74495c69f8-72m4k\" (UID: \"31534654-7557-4a65-b042-00d57f87e2dd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:27:42.683092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.683050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kfl\" (UniqueName: \"kubernetes.io/projected/31534654-7557-4a65-b042-00d57f87e2dd-kube-api-access-d9kfl\") pod \"limitador-operator-controller-manager-74495c69f8-72m4k\" (UID: \"31534654-7557-4a65-b042-00d57f87e2dd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:27:42.846619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:42.846578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:27:43.137849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:43.137823    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"]
Sep 12 00:27:43.139824 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:27:43.139799    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31534654_7557_4a65_b042_00d57f87e2dd.slice/crio-55c972ea98686d0492d009872fa69e3e6e4df223bb9620c35e62d19f5636f66a WatchSource:0}: Error finding container 55c972ea98686d0492d009872fa69e3e6e4df223bb9620c35e62d19f5636f66a: Status 404 returned error can't find the container with id 55c972ea98686d0492d009872fa69e3e6e4df223bb9620c35e62d19f5636f66a
Sep 12 00:27:43.141972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:43.141947    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:27:43.176526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:43.176497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:43.180288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:43.180260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k" event={"ID":"31534654-7557-4a65-b042-00d57f87e2dd","Type":"ContainerStarted","Data":"55c972ea98686d0492d009872fa69e3e6e4df223bb9620c35e62d19f5636f66a"}
Sep 12 00:27:43.497790 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:43.497696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:43.902949 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:43.902891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:46.747014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.746979    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:46.756458 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.756429    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"]
Sep 12 00:27:46.759082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.759065    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.762068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.762048    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d8v2s\""
Sep 12 00:27:46.786722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.786697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"]
Sep 12 00:27:46.805268 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.805230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3c66eb-5052-4e92-80b5-5f27ebc1116e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.805421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.805297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9779f\" (UniqueName: \"kubernetes.io/projected/da3c66eb-5052-4e92-80b5-5f27ebc1116e-kube-api-access-9779f\") pod \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.905830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.905795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3c66eb-5052-4e92-80b5-5f27ebc1116e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.906017 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.905858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9779f\" (UniqueName: \"kubernetes.io/projected/da3c66eb-5052-4e92-80b5-5f27ebc1116e-kube-api-access-9779f\") pod \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.906217 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.906193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3c66eb-5052-4e92-80b5-5f27ebc1116e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.915253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.915226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9779f\" (UniqueName: \"kubernetes.io/projected/da3c66eb-5052-4e92-80b5-5f27ebc1116e-kube-api-access-9779f\") pod \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:46.929672 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:46.929647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:47.068915 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:47.068882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:47.344985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:47.344957    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"]
Sep 12 00:27:47.346225 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:27:47.346196    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3c66eb_5052_4e92_80b5_5f27ebc1116e.slice/crio-30e009169729cc4a1f801157d295dc880807e8157822efe14ad5120808816f8a WatchSource:0}: Error finding container 30e009169729cc4a1f801157d295dc880807e8157822efe14ad5120808816f8a: Status 404 returned error can't find the container with id 30e009169729cc4a1f801157d295dc880807e8157822efe14ad5120808816f8a
Sep 12 00:27:47.506945 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:47.506909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:47.778143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:47.778010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:47.881646 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:47.881610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:48.203611 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:48.203571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr" event={"ID":"da3c66eb-5052-4e92-80b5-5f27ebc1116e","Type":"ContainerStarted","Data":"30e009169729cc4a1f801157d295dc880807e8157822efe14ad5120808816f8a"}
Sep 12 00:27:49.457296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:49.457258    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:50.109503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:50.109418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:50.721786 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:50.721706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:51.225556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.225228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k" event={"ID":"31534654-7557-4a65-b042-00d57f87e2dd","Type":"ContainerStarted","Data":"80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5"}
Sep 12 00:27:51.225556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.225331    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:27:51.242183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.242134    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k" podStartSLOduration=2.029762386 podStartE2EDuration="9.242114431s" totalImagesPullingTime="7.212352045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:27:51.240662489 +0000 UTC m=+674.177969010" watchObservedRunningTime="2026-09-12 00:27:51.242114431 +0000 UTC m=+674.179420952"
Sep 12 00:27:51.810838 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.810805    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"]
Sep 12 00:27:51.814193 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.814173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:27:51.816711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.816687    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 12 00:27:51.816832 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.816771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dljpg\""
Sep 12 00:27:51.826098 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.826078    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"]
Sep 12 00:27:51.952445 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:51.952419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcsv\" (UniqueName: \"kubernetes.io/projected/8a79b70d-f67d-4dc1-b8e1-89c66440c9f7-kube-api-access-wgcsv\") pod \"dns-operator-controller-manager-7b84cd7c9c-h7ptz\" (UID: \"8a79b70d-f67d-4dc1-b8e1-89c66440c9f7\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:27:52.053058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.053016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcsv\" (UniqueName: \"kubernetes.io/projected/8a79b70d-f67d-4dc1-b8e1-89c66440c9f7-kube-api-access-wgcsv\") pod \"dns-operator-controller-manager-7b84cd7c9c-h7ptz\" (UID: \"8a79b70d-f67d-4dc1-b8e1-89c66440c9f7\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:27:52.060831 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.060805    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcsv\" (UniqueName: \"kubernetes.io/projected/8a79b70d-f67d-4dc1-b8e1-89c66440c9f7-kube-api-access-wgcsv\") pod \"dns-operator-controller-manager-7b84cd7c9c-h7ptz\" (UID: \"8a79b70d-f67d-4dc1-b8e1-89c66440c9f7\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:27:52.124426 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.124386    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:27:52.232186 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.232149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr" event={"ID":"da3c66eb-5052-4e92-80b5-5f27ebc1116e","Type":"ContainerStarted","Data":"39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c"}
Sep 12 00:27:52.232371 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.232350    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:27:52.253788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.253730    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr" podStartSLOduration=1.718792622 podStartE2EDuration="6.253708744s" totalImagesPullingTime="4.534916122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:27:52.251868809 +0000 UTC m=+675.189175330" watchObservedRunningTime="2026-09-12 00:27:52.253708744 +0000 UTC m=+675.191015261"
Sep 12 00:27:52.395494 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:52.395397    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"]
Sep 12 00:27:52.397847 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:27:52.397816    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a79b70d_f67d_4dc1_b8e1_89c66440c9f7.slice/crio-8fc5e24cc4a9a90a21839c24b4867cd6343ec27fe24ee26a2c63f6314faafb78 WatchSource:0}: Error finding container 8fc5e24cc4a9a90a21839c24b4867cd6343ec27fe24ee26a2c63f6314faafb78: Status 404 returned error can't find the container with id 8fc5e24cc4a9a90a21839c24b4867cd6343ec27fe24ee26a2c63f6314faafb78
Sep 12 00:27:53.107320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:53.107291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:53.207716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:53.207685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:53.238146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:53.238115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz" event={"ID":"8a79b70d-f67d-4dc1-b8e1-89c66440c9f7","Type":"ContainerStarted","Data":"8fc5e24cc4a9a90a21839c24b4867cd6343ec27fe24ee26a2c63f6314faafb78"}
Sep 12 00:27:54.244613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.244583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz" event={"ID":"8a79b70d-f67d-4dc1-b8e1-89c66440c9f7","Type":"ContainerStarted","Data":"90c60757667ff93efbace9cf0b25d0834c1b824529773fa438542e4e787c0ffa"}
Sep 12 00:27:54.244950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.244691    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:27:54.270237 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.270156    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz" podStartSLOduration=1.5847487139999998 podStartE2EDuration="3.270137249s" totalImagesPullingTime="1.685388535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:27:54.268466508 +0000 UTC m=+677.205773029" watchObservedRunningTime="2026-09-12 00:27:54.270137249 +0000 UTC m=+677.207443767"
Sep 12 00:27:54.750605 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.750569    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"]
Sep 12 00:27:54.753922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.753900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.756572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.756548    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 12 00:27:54.756698 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.756610    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kshlz\""
Sep 12 00:27:54.756835 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.756811    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 12 00:27:54.757149 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.757121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:54.761130 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.760954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"]
Sep 12 00:27:54.878088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.878057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.878088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.878089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.878297 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.878111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k726\" (UniqueName: \"kubernetes.io/projected/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-kube-api-access-7k726\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.979199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.979166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.979199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.979199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.979418 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.979225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k726\" (UniqueName: \"kubernetes.io/projected/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-kube-api-access-7k726\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.979905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.979876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.981633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.981613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:54.993935 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.993906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:54.995629 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:54.995612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k726\" (UniqueName: \"kubernetes.io/projected/56571bde-c5fa-4aae-a2b2-67a1a5234cd5-kube-api-access-7k726\") pod \"kuadrant-console-plugin-fbdd4f445-tf8t9\" (UID: \"56571bde-c5fa-4aae-a2b2-67a1a5234cd5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:55.064300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:55.064274    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"
Sep 12 00:27:55.334553 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:55.334527    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9"]
Sep 12 00:27:55.336829 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:27:55.336795    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56571bde_c5fa_4aae_a2b2_67a1a5234cd5.slice/crio-8296efcf139f213d6bcdd017c76b155c3f56ca94939c7c9868e81a233ff34b90 WatchSource:0}: Error finding container 8296efcf139f213d6bcdd017c76b155c3f56ca94939c7c9868e81a233ff34b90: Status 404 returned error can't find the container with id 8296efcf139f213d6bcdd017c76b155c3f56ca94939c7c9868e81a233ff34b90
Sep 12 00:27:56.258546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:56.258505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9" event={"ID":"56571bde-c5fa-4aae-a2b2-67a1a5234cd5","Type":"ContainerStarted","Data":"8296efcf139f213d6bcdd017c76b155c3f56ca94939c7c9868e81a233ff34b90"}
Sep 12 00:27:56.904357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:56.904324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:56.951967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:56.951938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:57.323134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:57.323096    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:57.423392 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:57.423357    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:57.698706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:57.698630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:27:57.800193 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:27:57.800164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:00.281222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:00.281128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9" event={"ID":"56571bde-c5fa-4aae-a2b2-67a1a5234cd5","Type":"ContainerStarted","Data":"ecc79e236707ddaa0ff62a07b4277dec46af564d74e397f49c1ec6fcbbb0137c"}
Sep 12 00:28:00.298344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:00.298293    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tf8t9" podStartSLOduration=1.6917470140000002 podStartE2EDuration="6.298279349s" totalImagesPullingTime="4.606532335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:28:00.296628803 +0000 UTC m=+683.233935323" watchObservedRunningTime="2026-09-12 00:28:00.298279349 +0000 UTC m=+683.235585869"
Sep 12 00:28:02.234760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:02.234732    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:03.240983 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:03.240953    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:05.251855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.251825    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h7ptz"
Sep 12 00:28:05.579052 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.579018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:05.681527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.681491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:05.721119 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.721084    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"]
Sep 12 00:28:05.721344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.721321    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" containerName="manager" containerID="cri-o://39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c" gracePeriod=2
Sep 12 00:28:05.735925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.735896    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"]
Sep 12 00:28:05.753895 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.753855    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"da3c66eb-5052-4e92-80b5-5f27ebc1116e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-df8pr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:05.758270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.758247    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"]
Sep 12 00:28:05.758581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.758442    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k" podUID="31534654-7557-4a65-b042-00d57f87e2dd" containerName="manager" containerID="cri-o://80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5" gracePeriod=2
Sep 12 00:28:05.770223 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.770199    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"]
Sep 12 00:28:05.771980 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.771957    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"]
Sep 12 00:28:05.772421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.772399    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31534654-7557-4a65-b042-00d57f87e2dd" containerName="manager"
Sep 12 00:28:05.772552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.772424    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31534654-7557-4a65-b042-00d57f87e2dd" containerName="manager"
Sep 12 00:28:05.772552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.772463    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" containerName="manager"
Sep 12 00:28:05.772552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.772488    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" containerName="manager"
Sep 12 00:28:05.772722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.772574    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" containerName="manager"
Sep 12 00:28:05.772722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.772589    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31534654-7557-4a65-b042-00d57f87e2dd" containerName="manager"
Sep 12 00:28:05.775080 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.775064    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.777698 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.777670    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"31534654-7557-4a65-b042-00d57f87e2dd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-72m4k\": pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:05.781957 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.781931    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:05.786204 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.786171    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:05.788248 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.788220    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:05.793536 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.793509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"]
Sep 12 00:28:05.798526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.798505    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"]
Sep 12 00:28:05.801596 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.801579    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:05.817653 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.817630    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"]
Sep 12 00:28:05.818243 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.818216    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:05.820330 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.820295    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:05.872791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.872765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de1995f5-2f5b-4935-986e-d0e2cba95a31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2pdrf\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.872908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.872823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mrk\" (UniqueName: \"kubernetes.io/projected/de1995f5-2f5b-4935-986e-d0e2cba95a31-kube-api-access-k2mrk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2pdrf\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.973279 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.973247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de1995f5-2f5b-4935-986e-d0e2cba95a31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2pdrf\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.973423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.973284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/fd3180a5-2d82-4349-862b-ce5789e11ca1-kube-api-access-492ck\") pod \"limitador-operator-controller-manager-74495c69f8-n6q84\" (UID: \"fd3180a5-2d82-4349-862b-ce5789e11ca1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:05.973423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.973317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mrk\" (UniqueName: \"kubernetes.io/projected/de1995f5-2f5b-4935-986e-d0e2cba95a31-kube-api-access-k2mrk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2pdrf\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.973667 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.973615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de1995f5-2f5b-4935-986e-d0e2cba95a31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2pdrf\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.987385 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.987365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mrk\" (UniqueName: \"kubernetes.io/projected/de1995f5-2f5b-4935-986e-d0e2cba95a31-kube-api-access-k2mrk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2pdrf\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:05.996164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.996146    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:05.998450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:05.998432    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:05.998988 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:05.998965    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.001592 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.001568    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.003934 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.003914    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.006235 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.006216    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.073765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.073739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/fd3180a5-2d82-4349-862b-ce5789e11ca1-kube-api-access-492ck\") pod \"limitador-operator-controller-manager-74495c69f8-n6q84\" (UID: \"fd3180a5-2d82-4349-862b-ce5789e11ca1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:06.086945 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.086917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-492ck\" (UniqueName: \"kubernetes.io/projected/fd3180a5-2d82-4349-862b-ce5789e11ca1-kube-api-access-492ck\") pod \"limitador-operator-controller-manager-74495c69f8-n6q84\" (UID: \"fd3180a5-2d82-4349-862b-ce5789e11ca1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:06.164351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.164291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:06.172120 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.172102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:06.174376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.174344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9779f\" (UniqueName: \"kubernetes.io/projected/da3c66eb-5052-4e92-80b5-5f27ebc1116e-kube-api-access-9779f\") pod \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") "
Sep 12 00:28:06.174466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.174447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3c66eb-5052-4e92-80b5-5f27ebc1116e-extensions-socket-volume\") pod \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\" (UID: \"da3c66eb-5052-4e92-80b5-5f27ebc1116e\") "
Sep 12 00:28:06.174552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.174506    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9kfl\" (UniqueName: \"kubernetes.io/projected/31534654-7557-4a65-b042-00d57f87e2dd-kube-api-access-d9kfl\") pod \"31534654-7557-4a65-b042-00d57f87e2dd\" (UID: \"31534654-7557-4a65-b042-00d57f87e2dd\") "
Sep 12 00:28:06.175031 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.175003    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3c66eb-5052-4e92-80b5-5f27ebc1116e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "da3c66eb-5052-4e92-80b5-5f27ebc1116e" (UID: "da3c66eb-5052-4e92-80b5-5f27ebc1116e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:28:06.177018 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.176989    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3c66eb-5052-4e92-80b5-5f27ebc1116e-kube-api-access-9779f" (OuterVolumeSpecName: "kube-api-access-9779f") pod "da3c66eb-5052-4e92-80b5-5f27ebc1116e" (UID: "da3c66eb-5052-4e92-80b5-5f27ebc1116e"). InnerVolumeSpecName "kube-api-access-9779f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:28:06.177087 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.177032    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31534654-7557-4a65-b042-00d57f87e2dd-kube-api-access-d9kfl" (OuterVolumeSpecName: "kube-api-access-d9kfl") pod "31534654-7557-4a65-b042-00d57f87e2dd" (UID: "31534654-7557-4a65-b042-00d57f87e2dd"). InnerVolumeSpecName "kube-api-access-d9kfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:28:06.276050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.276025    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da3c66eb-5052-4e92-80b5-5f27ebc1116e-extensions-socket-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:28:06.276359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.276052    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kfl\" (UniqueName: \"kubernetes.io/projected/31534654-7557-4a65-b042-00d57f87e2dd-kube-api-access-d9kfl\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:28:06.276359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.276066    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9779f\" (UniqueName: \"kubernetes.io/projected/da3c66eb-5052-4e92-80b5-5f27ebc1116e-kube-api-access-9779f\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:28:06.311460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.311429    2562 generic.go:356] "Generic (PLEG): container finished" podID="31534654-7557-4a65-b042-00d57f87e2dd" containerID="80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5" exitCode=0
Sep 12 00:28:06.311568 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.311496    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.311568 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.311552    2562 scope.go:117] "RemoveContainer" containerID="80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5"
Sep 12 00:28:06.313109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.313082    2562 generic.go:356] "Generic (PLEG): container finished" podID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" containerID="39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c" exitCode=0
Sep 12 00:28:06.313202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.313130    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.313969 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.313924    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.316073 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.316044    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.317994 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.317971    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.319727 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.319703    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.321510 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.321465    2562 scope.go:117] "RemoveContainer" containerID="80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5"
Sep 12 00:28:06.321760 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.321741    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5\": container with ID starting with 80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5 not found: ID does not exist" containerID="80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5"
Sep 12 00:28:06.321823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.321768    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5"} err="failed to get container status \"80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5\": rpc error: code = NotFound desc = could not find container \"80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5\": container with ID starting with 80304e2743d59f3918dd1d91801d7a09695ec9b56efd7e09736c8195ee89ade5 not found: ID does not exist"
Sep 12 00:28:06.321823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.321789    2562 scope.go:117] "RemoveContainer" containerID="39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c"
Sep 12 00:28:06.323048 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.323028    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.324832 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.324805    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.326778 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.326755    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:06.328650 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.328629    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:06.329233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.329211    2562 scope.go:117] "RemoveContainer" containerID="39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c"
Sep 12 00:28:06.329495 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:06.329460    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c\": container with ID starting with 39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c not found: ID does not exist" containerID="39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c"
Sep 12 00:28:06.329559 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.329497    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c"} err="failed to get container status \"39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c\": rpc error: code = NotFound desc = could not find container \"39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c\": container with ID starting with 39b247925473ee0b2a5618c98b43fda157025508083b093f66028d1568b9404c not found: ID does not exist"
Sep 12 00:28:06.450334 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.450293    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"]
Sep 12 00:28:06.451543 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:28:06.451500    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3180a5_2d82_4349_862b_ce5789e11ca1.slice/crio-5df468475b0c2ca7ff7ba8d1868074d21a464ca8648c3187e67b36f119a23fe0 WatchSource:0}: Error finding container 5df468475b0c2ca7ff7ba8d1868074d21a464ca8648c3187e67b36f119a23fe0: Status 404 returned error can't find the container with id 5df468475b0c2ca7ff7ba8d1868074d21a464ca8648c3187e67b36f119a23fe0
Sep 12 00:28:06.452089 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:28:06.452002    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1995f5_2f5b_4935_986e_d0e2cba95a31.slice/crio-6eaffb23e95233a141cd650ed0b37f8d11d9eb748491625145fb4a2dd1e11bd3 WatchSource:0}: Error finding container 6eaffb23e95233a141cd650ed0b37f8d11d9eb748491625145fb4a2dd1e11bd3: Status 404 returned error can't find the container with id 6eaffb23e95233a141cd650ed0b37f8d11d9eb748491625145fb4a2dd1e11bd3
Sep 12 00:28:06.452089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:06.452063    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"]
Sep 12 00:28:07.319848 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.319812    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf" event={"ID":"de1995f5-2f5b-4935-986e-d0e2cba95a31","Type":"ContainerStarted","Data":"6e6d299cd2884588f94dd5ab40845e44502a6db603c68e5035697314cca155ab"}
Sep 12 00:28:07.320233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.319867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf" event={"ID":"de1995f5-2f5b-4935-986e-d0e2cba95a31","Type":"ContainerStarted","Data":"6eaffb23e95233a141cd650ed0b37f8d11d9eb748491625145fb4a2dd1e11bd3"}
Sep 12 00:28:07.320233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.319967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:07.321517 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.321488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84" event={"ID":"fd3180a5-2d82-4349-862b-ce5789e11ca1","Type":"ContainerStarted","Data":"93a4466cbf5df87aa11aa4b8d4897045e66a2e9d764f6a849ceaa0d644401223"}
Sep 12 00:28:07.321605 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.321554    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:07.321605 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.321582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84" event={"ID":"fd3180a5-2d82-4349-862b-ce5789e11ca1","Type":"ContainerStarted","Data":"5df468475b0c2ca7ff7ba8d1868074d21a464ca8648c3187e67b36f119a23fe0"}
Sep 12 00:28:07.322082 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:07.322056    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:07.323980 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:07.323954    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:07.349690 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.349621    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf" podStartSLOduration=2.349603484 podStartE2EDuration="2.349603484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:28:07.347401633 +0000 UTC m=+690.284708155" watchObservedRunningTime="2026-09-12 00:28:07.349603484 +0000 UTC m=+690.286910002"
Sep 12 00:28:07.349819 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:07.349650    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:07.351568 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:07.351543    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:07.379988 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.379943    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84" podStartSLOduration=2.379929365 podStartE2EDuration="2.379929365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:28:07.37859703 +0000 UTC m=+690.315903551" watchObservedRunningTime="2026-09-12 00:28:07.379929365 +0000 UTC m=+690.317235885"
Sep 12 00:28:07.567915 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:07.567881    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-72m4k\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="31534654-7557-4a65-b042-00d57f87e2dd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-72m4k"
Sep 12 00:28:07.569568 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.569542    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31534654-7557-4a65-b042-00d57f87e2dd" path="/var/lib/kubelet/pods/31534654-7557-4a65-b042-00d57f87e2dd/volumes"
Sep 12 00:28:07.569940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:07.569898    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" path="/var/lib/kubelet/pods/da3c66eb-5052-4e92-80b5-5f27ebc1116e/volumes"
Sep 12 00:28:07.570115 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:28:07.570072    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-df8pr\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="da3c66eb-5052-4e92-80b5-5f27ebc1116e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-df8pr"
Sep 12 00:28:08.002913 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:08.002834    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:08.101250 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:08.101218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:12.338277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:12.338242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:12.438825 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:12.438784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:16.153686 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:16.153653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:16.255115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:16.255084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:16.355139 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:16.355106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:16.409343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:16.409263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:17.344703 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:17.344642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:17.396320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:17.396291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:18.330561 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:18.330530    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:18.331002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:18.330979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n6q84"
Sep 12 00:28:21.912705 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:21.912669    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"]
Sep 12 00:28:21.913178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:21.912928    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf" podUID="de1995f5-2f5b-4935-986e-d0e2cba95a31" containerName="manager" containerID="cri-o://6e6d299cd2884588f94dd5ab40845e44502a6db603c68e5035697314cca155ab" gracePeriod=10
Sep 12 00:28:22.169813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.169752    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"]
Sep 12 00:28:22.186267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.186234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.188504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.188457    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"]
Sep 12 00:28:22.210110 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.210085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t8ch6\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.210241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.210128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwj2\" (UniqueName: \"kubernetes.io/projected/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-kube-api-access-ngwj2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t8ch6\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.311181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.311150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t8ch6\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.311312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.311184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwj2\" (UniqueName: \"kubernetes.io/projected/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-kube-api-access-ngwj2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t8ch6\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.311608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.311590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t8ch6\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.318715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.318694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwj2\" (UniqueName: \"kubernetes.io/projected/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-kube-api-access-ngwj2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t8ch6\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.394855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.394822    2562 generic.go:356] "Generic (PLEG): container finished" podID="de1995f5-2f5b-4935-986e-d0e2cba95a31" containerID="6e6d299cd2884588f94dd5ab40845e44502a6db603c68e5035697314cca155ab" exitCode=0
Sep 12 00:28:22.394998 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.394897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf" event={"ID":"de1995f5-2f5b-4935-986e-d0e2cba95a31","Type":"ContainerDied","Data":"6e6d299cd2884588f94dd5ab40845e44502a6db603c68e5035697314cca155ab"}
Sep 12 00:28:22.500424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.500354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:22.680462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.680440    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:22.714331 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.714299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de1995f5-2f5b-4935-986e-d0e2cba95a31-extensions-socket-volume\") pod \"de1995f5-2f5b-4935-986e-d0e2cba95a31\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") "
Sep 12 00:28:22.714514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.714342    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2mrk\" (UniqueName: \"kubernetes.io/projected/de1995f5-2f5b-4935-986e-d0e2cba95a31-kube-api-access-k2mrk\") pod \"de1995f5-2f5b-4935-986e-d0e2cba95a31\" (UID: \"de1995f5-2f5b-4935-986e-d0e2cba95a31\") "
Sep 12 00:28:22.714787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.714758    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1995f5-2f5b-4935-986e-d0e2cba95a31-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "de1995f5-2f5b-4935-986e-d0e2cba95a31" (UID: "de1995f5-2f5b-4935-986e-d0e2cba95a31"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:28:22.716737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.716715    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1995f5-2f5b-4935-986e-d0e2cba95a31-kube-api-access-k2mrk" (OuterVolumeSpecName: "kube-api-access-k2mrk") pod "de1995f5-2f5b-4935-986e-d0e2cba95a31" (UID: "de1995f5-2f5b-4935-986e-d0e2cba95a31"). InnerVolumeSpecName "kube-api-access-k2mrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:28:22.789263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.789238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"]
Sep 12 00:28:22.790858 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:28:22.790827    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80786198_cb1b_45b6_afc2_ccf3b7b5f3a8.slice/crio-9a29a9b2fc84cf3b66053a5eaca9f5e8c47d4b897881a09762057b95b9329ff3 WatchSource:0}: Error finding container 9a29a9b2fc84cf3b66053a5eaca9f5e8c47d4b897881a09762057b95b9329ff3: Status 404 returned error can't find the container with id 9a29a9b2fc84cf3b66053a5eaca9f5e8c47d4b897881a09762057b95b9329ff3
Sep 12 00:28:22.815401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.815377    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de1995f5-2f5b-4935-986e-d0e2cba95a31-extensions-socket-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:28:22.815488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:22.815405    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2mrk\" (UniqueName: \"kubernetes.io/projected/de1995f5-2f5b-4935-986e-d0e2cba95a31-kube-api-access-k2mrk\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:28:23.400867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.400837    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"
Sep 12 00:28:23.400867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.400853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf" event={"ID":"de1995f5-2f5b-4935-986e-d0e2cba95a31","Type":"ContainerDied","Data":"6eaffb23e95233a141cd650ed0b37f8d11d9eb748491625145fb4a2dd1e11bd3"}
Sep 12 00:28:23.401539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.400917    2562 scope.go:117] "RemoveContainer" containerID="6e6d299cd2884588f94dd5ab40845e44502a6db603c68e5035697314cca155ab"
Sep 12 00:28:23.402606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.402582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6" event={"ID":"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8","Type":"ContainerStarted","Data":"b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f"}
Sep 12 00:28:23.402718 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.402626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6" event={"ID":"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8","Type":"ContainerStarted","Data":"9a29a9b2fc84cf3b66053a5eaca9f5e8c47d4b897881a09762057b95b9329ff3"}
Sep 12 00:28:23.402718 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.402655    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:23.423971 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.423932    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6" podStartSLOduration=1.423917198 podStartE2EDuration="1.423917198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:28:23.421916683 +0000 UTC m=+706.359223203" watchObservedRunningTime="2026-09-12 00:28:23.423917198 +0000 UTC m=+706.361223717"
Sep 12 00:28:23.436938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.436916    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"]
Sep 12 00:28:23.442168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.442146    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2pdrf"]
Sep 12 00:28:23.570409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:23.570375    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1995f5-2f5b-4935-986e-d0e2cba95a31" path="/var/lib/kubelet/pods/de1995f5-2f5b-4935-986e-d0e2cba95a31/volumes"
Sep 12 00:28:26.555250 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:26.555217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:26.672344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:26.672309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:28.996322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:28.996292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:29.058423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:29.058370    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:34.411731 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:34.411654    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:28:38.288440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:38.288405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:38.476885 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:38.476839    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:44.984936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:44.984895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:54.492725 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:54.492688    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:56.353719 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.353684    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:56.452884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.452846    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:56.553031 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.552994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:56.735540 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.735455    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:28:56.735903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.735873    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1995f5-2f5b-4935-986e-d0e2cba95a31" containerName="manager"
Sep 12 00:28:56.735903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.735899    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1995f5-2f5b-4935-986e-d0e2cba95a31" containerName="manager"
Sep 12 00:28:56.736109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.736011    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1995f5-2f5b-4935-986e-d0e2cba95a31" containerName="manager"
Sep 12 00:28:56.739003 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.738982    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:56.741507 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.741486    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 12 00:28:56.746325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.746300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:28:56.770501 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.770462    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:28:56.896842 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.896814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce17ca8a-f51c-4ca3-a856-9398789d4dca-config-file\") pod \"limitador-limitador-84994f7ffc-t5k2j\" (UID: \"ce17ca8a-f51c-4ca3-a856-9398789d4dca\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:56.896998 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.896849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrm4x\" (UniqueName: \"kubernetes.io/projected/ce17ca8a-f51c-4ca3-a856-9398789d4dca-kube-api-access-jrm4x\") pod \"limitador-limitador-84994f7ffc-t5k2j\" (UID: \"ce17ca8a-f51c-4ca3-a856-9398789d4dca\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:56.997391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.997319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce17ca8a-f51c-4ca3-a856-9398789d4dca-config-file\") pod \"limitador-limitador-84994f7ffc-t5k2j\" (UID: \"ce17ca8a-f51c-4ca3-a856-9398789d4dca\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:56.997391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.997353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrm4x\" (UniqueName: \"kubernetes.io/projected/ce17ca8a-f51c-4ca3-a856-9398789d4dca-kube-api-access-jrm4x\") pod \"limitador-limitador-84994f7ffc-t5k2j\" (UID: \"ce17ca8a-f51c-4ca3-a856-9398789d4dca\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:56.997952 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:56.997933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce17ca8a-f51c-4ca3-a856-9398789d4dca-config-file\") pod \"limitador-limitador-84994f7ffc-t5k2j\" (UID: \"ce17ca8a-f51c-4ca3-a856-9398789d4dca\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:57.004975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.004946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrm4x\" (UniqueName: \"kubernetes.io/projected/ce17ca8a-f51c-4ca3-a856-9398789d4dca-kube-api-access-jrm4x\") pod \"limitador-limitador-84994f7ffc-t5k2j\" (UID: \"ce17ca8a-f51c-4ca3-a856-9398789d4dca\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:57.049608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.049581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:57.184969 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.184936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:57.249813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.249734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:28:57.249944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.249838    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-mh69v"]
Sep 12 00:28:57.252277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.252258    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:28:57.254665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.254643    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tt7ln\""
Sep 12 00:28:57.260238 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.260214    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mh69v"]
Sep 12 00:28:57.315198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.315177    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:28:57.317637 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:28:57.317613    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce17ca8a_f51c_4ca3_a856_9398789d4dca.slice/crio-a617542c33e4a951e3961b0dec067dd58f240ee85b7b42d907862f6b8e4c141b WatchSource:0}: Error finding container a617542c33e4a951e3961b0dec067dd58f240ee85b7b42d907862f6b8e4c141b: Status 404 returned error can't find the container with id a617542c33e4a951e3961b0dec067dd58f240ee85b7b42d907862f6b8e4c141b
Sep 12 00:28:57.400465 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.400442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdrc\" (UniqueName: \"kubernetes.io/projected/540179de-697d-4bcb-894b-6024a3919b76-kube-api-access-4vdrc\") pod \"authorino-59c974bd48-mh69v\" (UID: \"540179de-697d-4bcb-894b-6024a3919b76\") " pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:28:57.501806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.501742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdrc\" (UniqueName: \"kubernetes.io/projected/540179de-697d-4bcb-894b-6024a3919b76-kube-api-access-4vdrc\") pod \"authorino-59c974bd48-mh69v\" (UID: \"540179de-697d-4bcb-894b-6024a3919b76\") " pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:28:57.510405 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.510378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdrc\" (UniqueName: \"kubernetes.io/projected/540179de-697d-4bcb-894b-6024a3919b76-kube-api-access-4vdrc\") pod \"authorino-59c974bd48-mh69v\" (UID: \"540179de-697d-4bcb-894b-6024a3919b76\") " pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:28:57.563257 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.563235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:28:57.577844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.577813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j" event={"ID":"ce17ca8a-f51c-4ca3-a856-9398789d4dca","Type":"ContainerStarted","Data":"a617542c33e4a951e3961b0dec067dd58f240ee85b7b42d907862f6b8e4c141b"}
Sep 12 00:28:57.837132 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:57.837095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mh69v"]
Sep 12 00:28:57.838813 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:28:57.838782    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod540179de_697d_4bcb_894b_6024a3919b76.slice/crio-60598826871b108a7faf6bad26586d7d10c9c2c2b640e6c760a70d75eb6069a0 WatchSource:0}: Error finding container 60598826871b108a7faf6bad26586d7d10c9c2c2b640e6c760a70d75eb6069a0: Status 404 returned error can't find the container with id 60598826871b108a7faf6bad26586d7d10c9c2c2b640e6c760a70d75eb6069a0
Sep 12 00:28:58.585865 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:58.585824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j" event={"ID":"ce17ca8a-f51c-4ca3-a856-9398789d4dca","Type":"ContainerStarted","Data":"cc1cbc1ac6af0c3e3239e3f19a70c0d4ce236d3a6615cec58e99b64bce5a5d9a"}
Sep 12 00:28:58.586327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:58.585903    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:28:58.587054 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:58.587025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mh69v" event={"ID":"540179de-697d-4bcb-894b-6024a3919b76","Type":"ContainerStarted","Data":"60598826871b108a7faf6bad26586d7d10c9c2c2b640e6c760a70d75eb6069a0"}
Sep 12 00:28:58.602370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:58.602316    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j" podStartSLOduration=1.8669907989999999 podStartE2EDuration="2.602301363s" totalImagesPullingTime="735.310564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:28:58.600596617 +0000 UTC m=+741.537903138" watchObservedRunningTime="2026-09-12 00:28:58.602301363 +0000 UTC m=+741.539607882"
Sep 12 00:28:59.592315 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:59.592280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mh69v" event={"ID":"540179de-697d-4bcb-894b-6024a3919b76","Type":"ContainerStarted","Data":"5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887"}
Sep 12 00:28:59.607074 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:28:59.607028    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-mh69v" podStartSLOduration=1.338304307 podStartE2EDuration="2.607013899s" totalImagesPullingTime="1.268709592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:28:59.605529841 +0000 UTC m=+742.542836362" watchObservedRunningTime="2026-09-12 00:28:59.607013899 +0000 UTC m=+742.544320418"
Sep 12 00:29:09.593391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:09.593363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"
Sep 12 00:29:14.521565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:14.521530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:14.619636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:14.619603    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:15.089174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:15.089132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:27.548622 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:27.548585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:28.158316 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:28.158282    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:29.409245 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.409212    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mh69v"]
Sep 12 00:29:29.409741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.409466    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-mh69v" podUID="540179de-697d-4bcb-894b-6024a3919b76" containerName="authorino" containerID="cri-o://5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887" gracePeriod=30
Sep 12 00:29:29.652895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.652871    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:29:29.736723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.736635    2562 generic.go:356] "Generic (PLEG): container finished" podID="540179de-697d-4bcb-894b-6024a3919b76" containerID="5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887" exitCode=0
Sep 12 00:29:29.736723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.736685    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mh69v"
Sep 12 00:29:29.736920 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.736729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mh69v" event={"ID":"540179de-697d-4bcb-894b-6024a3919b76","Type":"ContainerDied","Data":"5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887"}
Sep 12 00:29:29.736920 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.736779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mh69v" event={"ID":"540179de-697d-4bcb-894b-6024a3919b76","Type":"ContainerDied","Data":"60598826871b108a7faf6bad26586d7d10c9c2c2b640e6c760a70d75eb6069a0"}
Sep 12 00:29:29.736920 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.736815    2562 scope.go:117] "RemoveContainer" containerID="5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887"
Sep 12 00:29:29.745105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.745090    2562 scope.go:117] "RemoveContainer" containerID="5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887"
Sep 12 00:29:29.745360 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:29:29.745343    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887\": container with ID starting with 5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887 not found: ID does not exist" containerID="5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887"
Sep 12 00:29:29.745407 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.745368    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887"} err="failed to get container status \"5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887\": rpc error: code = NotFound desc = could not find container \"5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887\": container with ID starting with 5dcded60bc0a046cd9df5df8f13b04918f3de831794d5ce18cbb7f2be6a8b887 not found: ID does not exist"
Sep 12 00:29:29.757746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.757725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vdrc\" (UniqueName: \"kubernetes.io/projected/540179de-697d-4bcb-894b-6024a3919b76-kube-api-access-4vdrc\") pod \"540179de-697d-4bcb-894b-6024a3919b76\" (UID: \"540179de-697d-4bcb-894b-6024a3919b76\") "
Sep 12 00:29:29.759607 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.759584    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540179de-697d-4bcb-894b-6024a3919b76-kube-api-access-4vdrc" (OuterVolumeSpecName: "kube-api-access-4vdrc") pod "540179de-697d-4bcb-894b-6024a3919b76" (UID: "540179de-697d-4bcb-894b-6024a3919b76"). InnerVolumeSpecName "kube-api-access-4vdrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:29:29.858757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:29.858730    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vdrc\" (UniqueName: \"kubernetes.io/projected/540179de-697d-4bcb-894b-6024a3919b76-kube-api-access-4vdrc\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:29:30.059985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:30.059959    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mh69v"]
Sep 12 00:29:30.066229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:30.066200    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mh69v"]
Sep 12 00:29:31.569740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:31.569708    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="540179de-697d-4bcb-894b-6024a3919b76" path="/var/lib/kubelet/pods/540179de-697d-4bcb-894b-6024a3919b76/volumes"
Sep 12 00:29:34.322084 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:34.322052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:34.407647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:34.407607    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:34.466556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:34.466520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:34.549967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:34.549932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:34.602244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:34.602167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:37.935273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:37.935242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:29:38.035589 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:29:38.035559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:09.438854 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:09.438765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:09.541233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:09.541197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:10.864661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:10.864625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:10.966118 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:10.966085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:11.388730 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:11.388699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:11.486414 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:11.486355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:11.564982 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:11.564940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:11.602521 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:11.602484    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:30:23.542220 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:23.542183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:23.641600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:23.641563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:31.077304 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:31.077272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:31.138697 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:31.138647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:32.109047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:32.109012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:35.921783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.920443    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"]
Sep 12 00:30:35.921783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.920978    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540179de-697d-4bcb-894b-6024a3919b76" containerName="authorino"
Sep 12 00:30:35.921783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.920998    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="540179de-697d-4bcb-894b-6024a3919b76" containerName="authorino"
Sep 12 00:30:35.921783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.921124    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="540179de-697d-4bcb-894b-6024a3919b76" containerName="authorino"
Sep 12 00:30:35.924365 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.924343    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:35.928021 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.927998    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 00:30:35.928021 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.928010    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 00:30:35.928190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.927996    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rjscr\""
Sep 12 00:30:35.928190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.927998    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 12 00:30:35.933280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:35.933249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"]
Sep 12 00:30:36.022750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.022718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.022929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.022766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prwx\" (UniqueName: \"kubernetes.io/projected/e6e4afa3-ef98-450c-bae3-e9da230afc8c-kube-api-access-2prwx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.022929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.022808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e4afa3-ef98-450c-bae3-e9da230afc8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.022929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.022833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.023064 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.022923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.023064 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.022970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prwx\" (UniqueName: \"kubernetes.io/projected/e6e4afa3-ef98-450c-bae3-e9da230afc8c-kube-api-access-2prwx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e4afa3-ef98-450c-bae3-e9da230afc8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.124908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.124880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.126784 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.126761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e4afa3-ef98-450c-bae3-e9da230afc8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.126952 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.126936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e4afa3-ef98-450c-bae3-e9da230afc8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.131859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.131830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prwx\" (UniqueName: \"kubernetes.io/projected/e6e4afa3-ef98-450c-bae3-e9da230afc8c-kube-api-access-2prwx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv\" (UID: \"e6e4afa3-ef98-450c-bae3-e9da230afc8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.236644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.236551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:36.366241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.366205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:36.495916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.495847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:30:36.507403 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:36.507381    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"]
Sep 12 00:30:36.509718 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:30:36.509689    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e4afa3_ef98_450c_bae3_e9da230afc8c.slice/crio-750c5df5532cf5d53efec08eb1697a890b1b697860b8928f35577a2fc009d848 WatchSource:0}: Error finding container 750c5df5532cf5d53efec08eb1697a890b1b697860b8928f35577a2fc009d848: Status 404 returned error can't find the container with id 750c5df5532cf5d53efec08eb1697a890b1b697860b8928f35577a2fc009d848
Sep 12 00:30:37.049696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:37.049655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv" event={"ID":"e6e4afa3-ef98-450c-bae3-e9da230afc8c","Type":"ContainerStarted","Data":"750c5df5532cf5d53efec08eb1697a890b1b697860b8928f35577a2fc009d848"}
Sep 12 00:30:38.056535 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:38.056418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv" event={"ID":"e6e4afa3-ef98-450c-bae3-e9da230afc8c","Type":"ContainerStarted","Data":"6657e91d7f87fa29794a677ab034b0d457951708795b520bcc4873aed5de78cb"}
Sep 12 00:30:38.073076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:38.073016    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv" podStartSLOduration=1.877202673 podStartE2EDuration="3.072999246s" totalImagesPullingTime="1.195796573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:30:38.071513506 +0000 UTC m=+841.008820027" watchObservedRunningTime="2026-09-12 00:30:38.072999246 +0000 UTC m=+841.010305762"
Sep 12 00:30:39.513654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:39.513612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:40.099941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:40.099899    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:30:42.840728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:42.840694    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:43.802648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:43.802616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:30:46.237741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:46.237703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:46.238181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:46.237791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:46.252024 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:46.251996    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:47.111010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:47.110983    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv"
Sep 12 00:30:53.409918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.409883    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"]
Sep 12 00:30:53.413703 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.413681    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.416067 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.416046    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 12 00:30:53.425174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.425149    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"]
Sep 12 00:30:53.442676 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.442648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:30:53.587150 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.587117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.587150 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.587150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.587402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.587233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.587402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.587301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.587402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.587337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvb9\" (UniqueName: \"kubernetes.io/projected/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-kube-api-access-kkvb9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.587402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.587369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688161 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688161 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvb9\" (UniqueName: \"kubernetes.io/projected/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-kube-api-access-kkvb9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688700 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.688757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.688711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.690447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.690428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.690865 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.690842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.695636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.695618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvb9\" (UniqueName: \"kubernetes.io/projected/aaa3efd0-a92a-4fc4-a28f-8526d4181c06-kube-api-access-kkvb9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rj8ts\" (UID: \"aaa3efd0-a92a-4fc4-a28f-8526d4181c06\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.724514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.724487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:30:53.996223 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:53.996195    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"]
Sep 12 00:30:53.997935 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:30:53.997908    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa3efd0_a92a_4fc4_a28f_8526d4181c06.slice/crio-ca025a1c74003329d58b967b6d7bc8a6390476595104f51223d810c5c4c115d7 WatchSource:0}: Error finding container ca025a1c74003329d58b967b6d7bc8a6390476595104f51223d810c5c4c115d7: Status 404 returned error can't find the container with id ca025a1c74003329d58b967b6d7bc8a6390476595104f51223d810c5c4c115d7
Sep 12 00:30:54.135625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:54.135594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts" event={"ID":"aaa3efd0-a92a-4fc4-a28f-8526d4181c06","Type":"ContainerStarted","Data":"ca025a1c74003329d58b967b6d7bc8a6390476595104f51223d810c5c4c115d7"}
Sep 12 00:30:54.600450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:54.600414    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:30:55.142483 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:55.142448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts" event={"ID":"aaa3efd0-a92a-4fc4-a28f-8526d4181c06","Type":"ContainerStarted","Data":"1be5b6c0ee1ac6022b02ae5be397eb0d7b5d58286b2ade2a8407a6f2d48ab089"}
Sep 12 00:30:55.163023 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:30:55.162965    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts" podStartSLOduration=1.987008686 podStartE2EDuration="2.162948327s" totalImagesPullingTime="175.939641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:30:55.161179541 +0000 UTC m=+858.098486077" watchObservedRunningTime="2026-09-12 00:30:55.162948327 +0000 UTC m=+858.100254846"
Sep 12 00:31:03.725535 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:03.725496    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:31:03.725995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:03.725565    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:31:03.738251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:03.738227    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:31:04.196214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:04.196184    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rj8ts"
Sep 12 00:31:05.349942 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:05.349908    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:31:06.696953 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:06.696913    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:31:07.221580 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:07.221546    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:31:12.938222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:12.938191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:31:14.598991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:14.598952    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:31:27.268712 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:27.268677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:31:32.715757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.715678    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"]
Sep 12 00:31:32.719040 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.719012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.721705 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.721683    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 12 00:31:32.728433 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.728406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"]
Sep 12 00:31:32.746354 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.746328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:31:32.820554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.820527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.820657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.820599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9f5\" (UniqueName: \"kubernetes.io/projected/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-kube-api-access-hk9f5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.820703 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.820654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.820752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.820700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.820752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.820734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.820830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.820797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922065 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9f5\" (UniqueName: \"kubernetes.io/projected/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-kube-api-access-hk9f5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922522 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922501    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.922599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.922578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.924422 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.924404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.924717 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.924699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:32.929893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:32.929874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9f5\" (UniqueName: \"kubernetes.io/projected/fd8e3e9e-09cb-44b0-80fa-56aa190b2cad-kube-api-access-hk9f5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn\" (UID: \"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:33.047491 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:33.047403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:33.315588 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:33.315561    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"]
Sep 12 00:31:33.317551 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:31:33.317528    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd8e3e9e_09cb_44b0_80fa_56aa190b2cad.slice/crio-87b5f33233d7bc25d67b9438c226889347ff1f5837e027088b6e99177a65e0db WatchSource:0}: Error finding container 87b5f33233d7bc25d67b9438c226889347ff1f5837e027088b6e99177a65e0db: Status 404 returned error can't find the container with id 87b5f33233d7bc25d67b9438c226889347ff1f5837e027088b6e99177a65e0db
Sep 12 00:31:33.331114 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:33.331085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn" event={"ID":"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad","Type":"ContainerStarted","Data":"87b5f33233d7bc25d67b9438c226889347ff1f5837e027088b6e99177a65e0db"}
Sep 12 00:31:34.338319 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:34.338283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn" event={"ID":"fd8e3e9e-09cb-44b0-80fa-56aa190b2cad","Type":"ContainerStarted","Data":"9c49340e0b184c2b9ef74d426b4776c2159f0ed9e5df6b3a4f849bfb8008de4e"}
Sep 12 00:31:34.355342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:34.355291    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn" podStartSLOduration=2.190052075 podStartE2EDuration="2.355276022s" totalImagesPullingTime="165.223947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:31:34.354188587 +0000 UTC m=+897.291495108" watchObservedRunningTime="2026-09-12 00:31:34.355276022 +0000 UTC m=+897.292582542"
Sep 12 00:31:34.801836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:34.801744    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:31:37.494892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:37.494858    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:31:37.507989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:37.507959    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:31:43.048314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:43.048284    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:43.048813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:43.048349    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:43.060804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:43.060780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:31:43.392224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:31:43.392195    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn"
Sep 12 00:32:58.024243 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:32:58.024200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:32:59.129152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:32:59.129112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:32:59.229706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:32:59.229673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:32:59.329968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:32:59.329937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.056981 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.056900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.160824 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.160778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.258182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.258135    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.417948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.417917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.507685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.507650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.586532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.586502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.686493 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.686395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.788221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.788191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:15.849429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:15.849400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:16.504607 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:16.504573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:16.566288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:16.566255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:17.016549 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.016513    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:17.093183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.093150    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7894-d74abb"]
Sep 12 00:33:17.095450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.095435    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:17.097830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.097808    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qwc4m\""
Sep 12 00:33:17.103287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.103256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7894-d74abb"]
Sep 12 00:33:17.218947 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.218913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7kp\" (UniqueName: \"kubernetes.io/projected/af622e93-c27c-4be9-8b63-635c03056290-kube-api-access-rk7kp\") pod \"test-curl-7894-d74abb\" (UID: \"af622e93-c27c-4be9-8b63-635c03056290\") " pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:17.319631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.319606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7kp\" (UniqueName: \"kubernetes.io/projected/af622e93-c27c-4be9-8b63-635c03056290-kube-api-access-rk7kp\") pod \"test-curl-7894-d74abb\" (UID: \"af622e93-c27c-4be9-8b63-635c03056290\") " pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:17.327739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.327718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7kp\" (UniqueName: \"kubernetes.io/projected/af622e93-c27c-4be9-8b63-635c03056290-kube-api-access-rk7kp\") pod \"test-curl-7894-d74abb\" (UID: \"af622e93-c27c-4be9-8b63-635c03056290\") " pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:17.405876 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.405800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:17.673176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.673142    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7894-d74abb"]
Sep 12 00:33:17.675073 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:17.675047    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf622e93_c27c_4be9_8b63_635c03056290.slice/crio-96665dbca8371543300446441ffaa8b00a484e3addfceba2cc3cfbb77d7f8036 WatchSource:0}: Error finding container 96665dbca8371543300446441ffaa8b00a484e3addfceba2cc3cfbb77d7f8036: Status 404 returned error can't find the container with id 96665dbca8371543300446441ffaa8b00a484e3addfceba2cc3cfbb77d7f8036
Sep 12 00:33:17.676821 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.676800    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:33:17.845005 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:17.844968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7894-d74abb" event={"ID":"af622e93-c27c-4be9-8b63-635c03056290","Type":"ContainerStarted","Data":"96665dbca8371543300446441ffaa8b00a484e3addfceba2cc3cfbb77d7f8036"}
Sep 12 00:33:18.506164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:18.506082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:18.578695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:18.578659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:18.665317 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:18.665273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:18.761000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:18.760904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:19.019912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:19.019827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:33:19.776524 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:19.776435    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:19.860275 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:19.860182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7894-d74abb" event={"ID":"af622e93-c27c-4be9-8b63-635c03056290","Type":"ContainerStarted","Data":"319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e"}
Sep 12 00:33:19.873492 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:19.873444    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:19.878341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:19.878292    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7894-d74abb" podStartSLOduration=0.949100396 podStartE2EDuration="2.878277425s" totalImagesPullingTime="1.929177029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:19.876665454 +0000 UTC m=+1002.813971975" watchObservedRunningTime="2026-09-12 00:33:19.878277425 +0000 UTC m=+1002.815583994"
Sep 12 00:33:20.204721 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.204638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:20.386020 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.385984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:20.470267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.470184    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7894-d74abb"]
Sep 12 00:33:20.472907 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.472877    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7894-d74abb"]
Sep 12 00:33:20.508355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.508329    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"]
Sep 12 00:33:20.508728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.508713    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af622e93-c27c-4be9-8b63-635c03056290" containerName="test-curl-7894-d74abb"
Sep 12 00:33:20.508780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.508730    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af622e93-c27c-4be9-8b63-635c03056290" containerName="test-curl-7894-d74abb"
Sep 12 00:33:20.508815 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.508792    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af622e93-c27c-4be9-8b63-635c03056290" containerName="test-curl-7894-d74abb"
Sep 12 00:33:20.511134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.511118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.514068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.514038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a4df66\""
Sep 12 00:33:20.514162 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.514047    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 00:33:20.514162 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.514047    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 00:33:20.515572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.515553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a4df66-dockercfg-kvg8g\""
Sep 12 00:33:20.529272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.529252    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"]
Sep 12 00:33:20.535264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.535208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:20.651063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.651023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9wg\" (UniqueName: \"kubernetes.io/projected/285c0202-812f-44c3-bff5-5f091280ea0f-kube-api-access-gk9wg\") pod \"payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.651235 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.651101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285c0202-812f-44c3-bff5-5f091280ea0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.751830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.751733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9wg\" (UniqueName: \"kubernetes.io/projected/285c0202-812f-44c3-bff5-5f091280ea0f-kube-api-access-gk9wg\") pod \"payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.751830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.751782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285c0202-812f-44c3-bff5-5f091280ea0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.752411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.752390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285c0202-812f-44c3-bff5-5f091280ea0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.759439 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.759413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9wg\" (UniqueName: \"kubernetes.io/projected/285c0202-812f-44c3-bff5-5f091280ea0f-kube-api-access-gk9wg\") pod \"payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.821437 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.821391    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:20.864907 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:33:20.864879    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7894-d74abb" container="test-curl-7894-d74abb"
Sep 12 00:33:20.880787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:20.880718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:21.090774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:21.090748    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"]
Sep 12 00:33:21.092419 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:21.092393    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285c0202_812f_44c3_bff5_5f091280ea0f.slice/crio-5e0c9923727135e38d8be419d820e72d28dca040de3455ed3bf75ecb01cccbd5 WatchSource:0}: Error finding container 5e0c9923727135e38d8be419d820e72d28dca040de3455ed3bf75ecb01cccbd5: Status 404 returned error can't find the container with id 5e0c9923727135e38d8be419d820e72d28dca040de3455ed3bf75ecb01cccbd5
Sep 12 00:33:21.659115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:21.659086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:21.772681 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:21.772643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:21.871214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:21.871180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp" event={"ID":"285c0202-812f-44c3-bff5-5f091280ea0f","Type":"ContainerStarted","Data":"5e0c9923727135e38d8be419d820e72d28dca040de3455ed3bf75ecb01cccbd5"}
Sep 12 00:33:21.871919 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:21.871300    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7894-d74abb" podUID="af622e93-c27c-4be9-8b63-635c03056290" containerName="test-curl-7894-d74abb" containerID="cri-o://319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e" gracePeriod=2
Sep 12 00:33:22.075684 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.075642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:22.278433 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.278392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:22.343431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.343400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:22.674159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.674085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:22.775413 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.775380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:22.877632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.877599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp" event={"ID":"285c0202-812f-44c3-bff5-5f091280ea0f","Type":"ContainerStarted","Data":"769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b"}
Sep 12 00:33:22.877990 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.877709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:22.895582 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:22.895535    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp" podStartSLOduration=1.626746297 podStartE2EDuration="2.895520972s" totalImagesPullingTime="1.268774675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:22.893575348 +0000 UTC m=+1005.830881879" watchObservedRunningTime="2026-09-12 00:33:22.895520972 +0000 UTC m=+1005.832827492"
Sep 12 00:33:23.059083 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.059002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:23.165488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.165438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:23.240410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.240365    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:23.274540 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.274509    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"]
Sep 12 00:33:23.276889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.276873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.279557 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.279466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-139288-dockercfg-4h9kc\""
Sep 12 00:33:23.279557 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.279520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-139288\""
Sep 12 00:33:23.285968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.285945    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"]
Sep 12 00:33:23.310685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.310622    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"]
Sep 12 00:33:23.312862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.312846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.322212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.322186    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"]
Sep 12 00:33:23.378222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.378186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwm7t\" (UniqueName: \"kubernetes.io/projected/42d2d637-9abe-4449-9efe-fbf4305af8d0-kube-api-access-hwm7t\") pod \"payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.378390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.378313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2d637-9abe-4449-9efe-fbf4305af8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.479356 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.479320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2d637-9abe-4449-9efe-fbf4305af8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.479577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.479378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwm7t\" (UniqueName: \"kubernetes.io/projected/42d2d637-9abe-4449-9efe-fbf4305af8d0-kube-api-access-hwm7t\") pod \"payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.479577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.479424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc72q\" (UniqueName: \"kubernetes.io/projected/fdc4378e-2441-4fe1-a9ac-77a1d595e507-kube-api-access-lc72q\") pod \"payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.479577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.479494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc4378e-2441-4fe1-a9ac-77a1d595e507-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.480321 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.480296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2d637-9abe-4449-9efe-fbf4305af8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.487386 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.487351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwm7t\" (UniqueName: \"kubernetes.io/projected/42d2d637-9abe-4449-9efe-fbf4305af8d0-kube-api-access-hwm7t\") pod \"payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.580175 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.580137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc72q\" (UniqueName: \"kubernetes.io/projected/fdc4378e-2441-4fe1-a9ac-77a1d595e507-kube-api-access-lc72q\") pod \"payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.580359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.580204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc4378e-2441-4fe1-a9ac-77a1d595e507-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.580935 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.580913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc4378e-2441-4fe1-a9ac-77a1d595e507-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.587867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.587834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc72q\" (UniqueName: \"kubernetes.io/projected/fdc4378e-2441-4fe1-a9ac-77a1d595e507-kube-api-access-lc72q\") pod \"payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.588000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.587976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:23.625346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.625298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:23.765847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.765814    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:23.829834 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.829800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:23.863019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.862992    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"]
Sep 12 00:33:23.866054 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:23.866026    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d2d637_9abe_4449_9efe_fbf4305af8d0.slice/crio-ddee3e3bd406d2d66e77e431f560cd87ac5897b14e1366e9511c77df07a1f463 WatchSource:0}: Error finding container ddee3e3bd406d2d66e77e431f560cd87ac5897b14e1366e9511c77df07a1f463: Status 404 returned error can't find the container with id ddee3e3bd406d2d66e77e431f560cd87ac5897b14e1366e9511c77df07a1f463
Sep 12 00:33:23.888237 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.888206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85" event={"ID":"42d2d637-9abe-4449-9efe-fbf4305af8d0","Type":"ContainerStarted","Data":"ddee3e3bd406d2d66e77e431f560cd87ac5897b14e1366e9511c77df07a1f463"}
Sep 12 00:33:23.899634 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.899613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:23.910989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:23.910968    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"]
Sep 12 00:33:23.932909 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:23.932870    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc4378e_2441_4fe1_a9ac_77a1d595e507.slice/crio-72a1709c96c307aa810aaa735c13bbfac917ce90f8fe6fd531a471f1ec0260af WatchSource:0}: Error finding container 72a1709c96c307aa810aaa735c13bbfac917ce90f8fe6fd531a471f1ec0260af: Status 404 returned error can't find the container with id 72a1709c96c307aa810aaa735c13bbfac917ce90f8fe6fd531a471f1ec0260af
Sep 12 00:33:24.043260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.043235    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:24.086719 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.086681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:24.183878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.183791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:24.185962 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.185936    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk7kp\" (UniqueName: \"kubernetes.io/projected/af622e93-c27c-4be9-8b63-635c03056290-kube-api-access-rk7kp\") pod \"af622e93-c27c-4be9-8b63-635c03056290\" (UID: \"af622e93-c27c-4be9-8b63-635c03056290\") "
Sep 12 00:33:24.188292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.188259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af622e93-c27c-4be9-8b63-635c03056290-kube-api-access-rk7kp" (OuterVolumeSpecName: "kube-api-access-rk7kp") pod "af622e93-c27c-4be9-8b63-635c03056290" (UID: "af622e93-c27c-4be9-8b63-635c03056290"). InnerVolumeSpecName "kube-api-access-rk7kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:33:24.287816 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.287717    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk7kp\" (UniqueName: \"kubernetes.io/projected/af622e93-c27c-4be9-8b63-635c03056290-kube-api-access-rk7kp\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:33:24.312260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.312229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:24.416898 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.416869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:24.629267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.629228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:24.726982 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.726950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:24.895146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.895105    2562 generic.go:356] "Generic (PLEG): container finished" podID="af622e93-c27c-4be9-8b63-635c03056290" containerID="319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e" exitCode=137
Sep 12 00:33:24.895146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.895150    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:24.895663 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.895207    2562 scope.go:117] "RemoveContainer" containerID="319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e"
Sep 12 00:33:24.897053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.897027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85" event={"ID":"42d2d637-9abe-4449-9efe-fbf4305af8d0","Type":"ContainerStarted","Data":"0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25"}
Sep 12 00:33:24.897202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.897117    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:24.898859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.898837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h" event={"ID":"fdc4378e-2441-4fe1-a9ac-77a1d595e507","Type":"ContainerStarted","Data":"f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172"}
Sep 12 00:33:24.898956 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.898872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h" event={"ID":"fdc4378e-2441-4fe1-a9ac-77a1d595e507","Type":"ContainerStarted","Data":"72a1709c96c307aa810aaa735c13bbfac917ce90f8fe6fd531a471f1ec0260af"}
Sep 12 00:33:24.898956 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.898921    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:24.905608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.905591    2562 scope.go:117] "RemoveContainer" containerID="319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e"
Sep 12 00:33:24.905910 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:33:24.905894    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e\": container with ID starting with 319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e not found: ID does not exist" containerID="319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e"
Sep 12 00:33:24.905983 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.905917    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e"} err="failed to get container status \"319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e\": rpc error: code = NotFound desc = could not find container \"319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e\": container with ID starting with 319d06e6df24aeebbb33de3d987741af07d199a93abface43332094eb9c4425e not found: ID does not exist"
Sep 12 00:33:24.913076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.913026    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85" podStartSLOduration=1.9130144489999998 podStartE2EDuration="1.913014449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:24.911713222 +0000 UTC m=+1007.849019743" watchObservedRunningTime="2026-09-12 00:33:24.913014449 +0000 UTC m=+1007.850320969"
Sep 12 00:33:24.913736 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:33:24.913706    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7894-d74abb\" is forbidden: User \"system:node:ip-10-0-140-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-187.ec2.internal' and this object" podUID="af622e93-c27c-4be9-8b63-635c03056290" pod="opendatahub/test-curl-7894-d74abb"
Sep 12 00:33:24.933571 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:24.933530    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h" podStartSLOduration=1.933514686 podStartE2EDuration="1.933514686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:24.931378955 +0000 UTC m=+1007.868685481" watchObservedRunningTime="2026-09-12 00:33:24.933514686 +0000 UTC m=+1007.870821206"
Sep 12 00:33:25.530431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:25.530396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:25.570363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:25.570334    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af622e93-c27c-4be9-8b63-635c03056290" path="/var/lib/kubelet/pods/af622e93-c27c-4be9-8b63-635c03056290/volumes"
Sep 12 00:33:26.346884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:26.346840    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:26.989224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:26.989190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:27.086237 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:27.086202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:27.239829 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:27.239745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:27.683619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:27.683590    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:27.767444 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:27.767408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:28.769724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:28.769683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:30.092814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:30.092779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:31.126295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:31.126264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:31.241058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:31.241026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:31.681840 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:31.681810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:31.810950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:31.810915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:31.908859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:31.908827    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:32.012966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.012888    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"]
Sep 12 00:33:32.017931 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.017910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.020175 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.020156    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4aaf1a\""
Sep 12 00:33:32.020285 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.020204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4aaf1a-dockercfg-mh8gp\""
Sep 12 00:33:32.026137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.026106    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"]
Sep 12 00:33:32.059683 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.059650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778bd95f-4b67-4c4a-84ce-4a9769598bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.059812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.059711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69gq\" (UniqueName: \"kubernetes.io/projected/778bd95f-4b67-4c4a-84ce-4a9769598bee-kube-api-access-p69gq\") pod \"payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.111344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.111308    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"]
Sep 12 00:33:32.114979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.114963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.124121 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.124095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"]
Sep 12 00:33:32.160916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.160874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778bd95f-4b67-4c4a-84ce-4a9769598bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.160916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.160912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplnv\" (UniqueName: \"kubernetes.io/projected/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-kube-api-access-hplnv\") pod \"payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") " pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.161415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.160972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-plugins-config-volume\") pod \"payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") " pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.161415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.161005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69gq\" (UniqueName: \"kubernetes.io/projected/778bd95f-4b67-4c4a-84ce-4a9769598bee-kube-api-access-p69gq\") pod \"payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.161691 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.161670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778bd95f-4b67-4c4a-84ce-4a9769598bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.168151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.168124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69gq\" (UniqueName: \"kubernetes.io/projected/778bd95f-4b67-4c4a-84ce-4a9769598bee-kube-api-access-p69gq\") pod \"payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.261781 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.261745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplnv\" (UniqueName: \"kubernetes.io/projected/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-kube-api-access-hplnv\") pod \"payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") " pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.261956 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.261819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-plugins-config-volume\") pod \"payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") " pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.262572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.262553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-plugins-config-volume\") pod \"payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") " pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.269399 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.269337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplnv\" (UniqueName: \"kubernetes.io/projected/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-kube-api-access-hplnv\") pod \"payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") " pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.329806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.329773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.394452 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.394417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:32.426386 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.426360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.458183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.458154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:32.601165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.601139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"]
Sep 12 00:33:32.603412 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:32.603379    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778bd95f_4b67_4c4a_84ce_4a9769598bee.slice/crio-fb55df784a51ebe21164aee1c8679fbac000ef24f1c5235a101489611ab88b9d WatchSource:0}: Error finding container fb55df784a51ebe21164aee1c8679fbac000ef24f1c5235a101489611ab88b9d: Status 404 returned error can't find the container with id fb55df784a51ebe21164aee1c8679fbac000ef24f1c5235a101489611ab88b9d
Sep 12 00:33:32.701817 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.701779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"]
Sep 12 00:33:32.703676 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:32.703647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdea681_0219_4b4e_8c7f_3cc8f4d18520.slice/crio-e9cfba904a70c1f4aee67b119347089ec0b25b7ad417f0313e247ed092d1fa30 WatchSource:0}: Error finding container e9cfba904a70c1f4aee67b119347089ec0b25b7ad417f0313e247ed092d1fa30: Status 404 returned error can't find the container with id e9cfba904a70c1f4aee67b119347089ec0b25b7ad417f0313e247ed092d1fa30
Sep 12 00:33:32.720230 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.720199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:32.941341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.941243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf" event={"ID":"778bd95f-4b67-4c4a-84ce-4a9769598bee","Type":"ContainerStarted","Data":"b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8"}
Sep 12 00:33:32.941341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.941301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf" event={"ID":"778bd95f-4b67-4c4a-84ce-4a9769598bee","Type":"ContainerStarted","Data":"fb55df784a51ebe21164aee1c8679fbac000ef24f1c5235a101489611ab88b9d"}
Sep 12 00:33:32.941341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.941335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:32.943017 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.942985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx" event={"ID":"4bdea681-0219-4b4e-8c7f-3cc8f4d18520","Type":"ContainerStarted","Data":"87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921"}
Sep 12 00:33:32.943164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.943029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx" event={"ID":"4bdea681-0219-4b4e-8c7f-3cc8f4d18520","Type":"ContainerStarted","Data":"e9cfba904a70c1f4aee67b119347089ec0b25b7ad417f0313e247ed092d1fa30"}
Sep 12 00:33:32.943164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.943049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:32.956655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.956607    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf" podStartSLOduration=1.956592987 podStartE2EDuration="1.956592987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:32.955767139 +0000 UTC m=+1015.893073659" watchObservedRunningTime="2026-09-12 00:33:32.956592987 +0000 UTC m=+1015.893899515"
Sep 12 00:33:32.972738 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:32.972685    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx" podStartSLOduration=0.972666671 podStartE2EDuration="972.666671ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:32.971753733 +0000 UTC m=+1015.909060256" watchObservedRunningTime="2026-09-12 00:33:32.972666671 +0000 UTC m=+1015.909973192"
Sep 12 00:33:33.038249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.038220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:33.139390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.139350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:33.284562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.284467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:33.727388 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.727354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:33.833289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.833258    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:33.890435 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.890397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:33:33.902320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:33.902291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:34.677943 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:34.677907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:35.552906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:35.552871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:35.801150 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:35.801112    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:33:35.861357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:35.861320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:35.907105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:35.907080    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:33:35.907388 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:35.907371    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:33:36.034393 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:36.034360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:36.136547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:36.136450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:36.242113 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:36.242080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:36.741081 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:36.741041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:37.735737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:37.735700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:38.379868 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:38.379835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:38.445580 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:38.445544    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:38.541218 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:38.541183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:38.743539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:38.743427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:38.893425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:38.893389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:39.031813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:39.031737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:39.124343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:39.124310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:39.183101 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:39.183062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:39.580322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:39.580289    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:39.913784 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:39.913706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:40.674235 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:40.674198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:40.814993 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:40.814956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:41.575085 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:41.575053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:42.614332 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:42.614295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:42.820415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:42.820369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:42.914387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:42.914308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:43.081677 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:43.081640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:43.206303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:43.206226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:43.309804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:43.309772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:43.952958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:43.952927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:33:43.953370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:43.953001    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:33:44.034271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:44.034240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:45.035163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.035127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:45.825082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.825035    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"]
Sep 12 00:33:45.830004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.829981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.832462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.832437    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-39d42b-serving-cert\""
Sep 12 00:33:45.832625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.832538    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 00:33:45.832625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.832563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 00:33:45.832798 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.832781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:33:45.833503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.833459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f4scw\""
Sep 12 00:33:45.837918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.837895    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"]
Sep 12 00:33:45.859915 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.859884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:45.889962 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.889932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-metrics-certs\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.890096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.890031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.890096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.890059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.890176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.890104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdsh\" (UniqueName: \"kubernetes.io/projected/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-kube-api-access-vzdsh\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.957188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.957156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:45.991402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.991371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.991402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.991404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.991631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.991430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdsh\" (UniqueName: \"kubernetes.io/projected/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-kube-api-access-vzdsh\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.991631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.991502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-metrics-certs\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.991631 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:33:45.991553    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-serving-cert: secret "maas-api-e2e-inf-b-39d42b-serving-cert" not found
Sep 12 00:33:45.991746 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:33:45.991642    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls podName:36a847b9-db0b-4d3f-8b1c-d2b2d33aee65 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:46.491622138 +0000 UTC m=+1029.428928644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls") pod "maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" (UID: "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65") : secret "maas-api-e2e-inf-b-39d42b-serving-cert" not found
Sep 12 00:33:45.992149 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.992128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:45.993824 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:45.993807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-metrics-certs\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:46.002130 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.002108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdsh\" (UniqueName: \"kubernetes.io/projected/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-kube-api-access-vzdsh\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:46.307256 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.307168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:46.405788 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.405750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:46.411715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.411685    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"]
Sep 12 00:33:46.415372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.415355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.417878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.417858    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39d42b\""
Sep 12 00:33:46.417958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.417858    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39d42b-dockercfg-kbfgg\""
Sep 12 00:33:46.424583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.424558    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"]
Sep 12 00:33:46.473688 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.473655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:46.496371 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.496334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:46.496564 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.496379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ede5fa-47c9-4e3b-b2db-e09356a32b65-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.496564 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.496443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz9lf\" (UniqueName: \"kubernetes.io/projected/96ede5fa-47c9-4e3b-b2db-e09356a32b65-kube-api-access-zz9lf\") pod \"payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.498707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.498679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls\") pod \"maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:46.597765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.597726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ede5fa-47c9-4e3b-b2db-e09356a32b65-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.597940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.597814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz9lf\" (UniqueName: \"kubernetes.io/projected/96ede5fa-47c9-4e3b-b2db-e09356a32b65-kube-api-access-zz9lf\") pod \"payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.598457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.598435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ede5fa-47c9-4e3b-b2db-e09356a32b65-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.605747 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.605718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz9lf\" (UniqueName: \"kubernetes.io/projected/96ede5fa-47c9-4e3b-b2db-e09356a32b65-kube-api-access-zz9lf\") pod \"payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.683447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.683408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:46.727073 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.727045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:46.743042 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:46.743015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:47.002426 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:47.002364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"]
Sep 12 00:33:47.004622 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:47.004590    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ede5fa_47c9_4e3b_b2db_e09356a32b65.slice/crio-f8bb658126728ea50093d5dbe7e81228f7de8ec1a96e3310a5f9a256d20c1f7d WatchSource:0}: Error finding container f8bb658126728ea50093d5dbe7e81228f7de8ec1a96e3310a5f9a256d20c1f7d: Status 404 returned error can't find the container with id f8bb658126728ea50093d5dbe7e81228f7de8ec1a96e3310a5f9a256d20c1f7d
Sep 12 00:33:47.027930 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:47.027902    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"]
Sep 12 00:33:47.029142 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:47.029101    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a847b9_db0b_4d3f_8b1c_d2b2d33aee65.slice/crio-6bb5f95e68bd512801320d956c9de2f7b7817b219f23c78e316378b6b4c74bea WatchSource:0}: Error finding container 6bb5f95e68bd512801320d956c9de2f7b7817b219f23c78e316378b6b4c74bea: Status 404 returned error can't find the container with id 6bb5f95e68bd512801320d956c9de2f7b7817b219f23c78e316378b6b4c74bea
Sep 12 00:33:47.031370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:47.031342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76" event={"ID":"96ede5fa-47c9-4e3b-b2db-e09356a32b65","Type":"ContainerStarted","Data":"f8bb658126728ea50093d5dbe7e81228f7de8ec1a96e3310a5f9a256d20c1f7d"}
Sep 12 00:33:47.432139 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:47.432104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:47.720637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:47.720527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:48.045707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:48.045617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" event={"ID":"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65","Type":"ContainerStarted","Data":"6bb5f95e68bd512801320d956c9de2f7b7817b219f23c78e316378b6b4c74bea"}
Sep 12 00:33:48.048077 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:48.047904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76" event={"ID":"96ede5fa-47c9-4e3b-b2db-e09356a32b65","Type":"ContainerStarted","Data":"330a6de610bea5401f879a0aa2e7299d8a34d671b74c57973548cb45ede14b8d"}
Sep 12 00:33:48.048205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:48.048121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:33:48.069895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:48.067356    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76" podStartSLOduration=2.067339576 podStartE2EDuration="2.067339576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:48.066373393 +0000 UTC m=+1031.003679914" watchObservedRunningTime="2026-09-12 00:33:48.067339576 +0000 UTC m=+1031.004646096"
Sep 12 00:33:48.386202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:48.386163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:49.138440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:49.138402    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:49.251393 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:49.251339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:49.342616 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:49.342580    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:49.485757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:49.485727    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:49.955309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:49.955273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:50.061765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:50.061730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" event={"ID":"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65","Type":"ContainerStarted","Data":"f1a378cae17c10cf51967bfe6aa0082862526e2fe7f977dd2c470fec4ea2d975"}
Sep 12 00:33:50.061952 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:50.061885    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:50.079265 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:50.079220    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" podStartSLOduration=2.678002923 podStartE2EDuration="5.079204411s" totalImagesPullingTime="2.401201488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:50.078309538 +0000 UTC m=+1033.015616058" watchObservedRunningTime="2026-09-12 00:33:50.079204411 +0000 UTC m=+1033.016510960"
Sep 12 00:33:50.712632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:50.712600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:51.439646 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:51.439609    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:51.502835 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:51.502793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:33:52.235826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.235794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:52.537100 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.537008    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"]
Sep 12 00:33:52.540916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.540891    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:33:52.543404 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.543381    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:33:52.547618 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.547591    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"]
Sep 12 00:33:52.658328 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.658283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd\" (UID: \"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:33:52.758944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.758911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd\" (UID: \"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:33:52.759648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.759627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd\" (UID: \"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:33:52.855110 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:52.855065    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:33:53.125695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:53.125671    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"]
Sep 12 00:33:53.127140 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:33:53.127113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fb79d0_60e6_47f3_8266_b1ca5c8b40a6.slice/crio-63445e827955ddbd958183e4fc98e910ddbbe805f030d8e112e1fc6cdd58bb82 WatchSource:0}: Error finding container 63445e827955ddbd958183e4fc98e910ddbbe805f030d8e112e1fc6cdd58bb82: Status 404 returned error can't find the container with id 63445e827955ddbd958183e4fc98e910ddbbe805f030d8e112e1fc6cdd58bb82
Sep 12 00:33:53.249697 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:53.249664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:53.886886 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:53.886842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:54.085316 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:54.085278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd" event={"ID":"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6","Type":"ContainerStarted","Data":"63445e827955ddbd958183e4fc98e910ddbbe805f030d8e112e1fc6cdd58bb82"}
Sep 12 00:33:54.741310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:54.741280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:55.459904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:55.459862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:55.574521 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:55.574485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:55.675368 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:55.675328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:56.072635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:56.072604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:33:56.234293 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:56.234251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:57.060423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:57.060386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:57.710013 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:57.709981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:58.443322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:58.443292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:33:59.056830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:33:59.056798    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:34:06.856746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:06.856699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:08.140543 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:08.140509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:08.764903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:08.764872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:09.163335 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:09.163243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd" event={"ID":"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6","Type":"ContainerStarted","Data":"60b412f45d4dca9ed9d8fc441ced5c0615be6f62eba314b5642b1d33c5f0dff6"}
Sep 12 00:34:09.178294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:09.178240    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd" podStartSLOduration=1.737695305 podStartE2EDuration="17.17822358s" totalImagesPullingTime="15.440528275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:34:09.174654231 +0000 UTC m=+1052.111960751" watchObservedRunningTime="2026-09-12 00:34:09.17822358 +0000 UTC m=+1052.115530099"
Sep 12 00:34:09.559133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:09.559052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:10.170237 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.170205    2562 generic.go:356] "Generic (PLEG): container finished" podID="71fb79d0-60e6-47f3-8266-b1ca5c8b40a6" containerID="60b412f45d4dca9ed9d8fc441ced5c0615be6f62eba314b5642b1d33c5f0dff6" exitCode=0
Sep 12 00:34:10.170632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.170265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd" event={"ID":"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6","Type":"ContainerDied","Data":"60b412f45d4dca9ed9d8fc441ced5c0615be6f62eba314b5642b1d33c5f0dff6"}
Sep 12 00:34:10.188432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.188403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:10.200023 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.199991    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"]
Sep 12 00:34:10.203898 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.203879    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.206604 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.206579    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3c466c\"/\"openshift-service-ca.crt\""
Sep 12 00:34:10.206604 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.206596    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c466c\"/\"default-dockercfg-m5sdb\""
Sep 12 00:34:10.206814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.206650    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c466c\"/\"simulator-w0-3c466c-kserve-self-signed-certs\""
Sep 12 00:34:10.206814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.206596    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3c466c\"/\"kube-root-ca.crt\""
Sep 12 00:34:10.214224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.214191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"]
Sep 12 00:34:10.217889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.217863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-model-cache\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.218085 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.218064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-tmp-dir\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.218214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.218198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-dshm\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.218341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.218327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-home\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.218727 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.218591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc8359-4a2c-44c1-b387-5840a02b1c48-tls-certs\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.218727 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.218675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqws\" (UniqueName: \"kubernetes.io/projected/62dc8359-4a2c-44c1-b387-5840a02b1c48-kube-api-access-2gqws\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.306928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.306887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:10.319957 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.319930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-model-cache\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.319963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-tmp-dir\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.319981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-dshm\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.320092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-home\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320285 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.320160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc8359-4a2c-44c1-b387-5840a02b1c48-tls-certs\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320285 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.320252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqws\" (UniqueName: \"kubernetes.io/projected/62dc8359-4a2c-44c1-b387-5840a02b1c48-kube-api-access-2gqws\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320392 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.320318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-tmp-dir\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320444 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.320397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-model-cache\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.320527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.320504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-home\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.322288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.322264    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-dshm\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.322518 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.322496    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc8359-4a2c-44c1-b387-5840a02b1c48-tls-certs\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.328028 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.328007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqws\" (UniqueName: \"kubernetes.io/projected/62dc8359-4a2c-44c1-b387-5840a02b1c48-kube-api-access-2gqws\") pod \"simulator-w0-3c466c-kserve-5c59b76644-gzdp6\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") " pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.523615 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.523529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:10.792779 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.792754    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"]
Sep 12 00:34:10.794987 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:34:10.794952    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62dc8359_4a2c_44c1_b387_5840a02b1c48.slice/crio-8d50d2b2df491556c286bbf5feaa8d10acab25b1a43f4935c23a700cbf87b1aa WatchSource:0}: Error finding container 8d50d2b2df491556c286bbf5feaa8d10acab25b1a43f4935c23a700cbf87b1aa: Status 404 returned error can't find the container with id 8d50d2b2df491556c286bbf5feaa8d10acab25b1a43f4935c23a700cbf87b1aa
Sep 12 00:34:10.844170 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:10.844143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:11.177263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.177226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6" event={"ID":"62dc8359-4a2c-44c1-b387-5840a02b1c48","Type":"ContainerStarted","Data":"d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987"}
Sep 12 00:34:11.177707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.177277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6" event={"ID":"62dc8359-4a2c-44c1-b387-5840a02b1c48","Type":"ContainerStarted","Data":"8d50d2b2df491556c286bbf5feaa8d10acab25b1a43f4935c23a700cbf87b1aa"}
Sep 12 00:34:11.198423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.198367    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6" podStartSLOduration=1.198351162 podStartE2EDuration="1.198351162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:34:11.196884776 +0000 UTC m=+1054.134191294" watchObservedRunningTime="2026-09-12 00:34:11.198351162 +0000 UTC m=+1054.135657682"
Sep 12 00:34:11.298638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.298610    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:34:11.330612 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.330584    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6-maas-api-service-ca\") pod \"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6\" (UID: \"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6\") "
Sep 12 00:34:11.330995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.330969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71fb79d0-60e6-47f3-8266-b1ca5c8b40a6" (UID: "71fb79d0-60e6-47f3-8266-b1ca5c8b40a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:34:11.431581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.431497    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:34:11.565378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:11.565339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:12.184194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:12.184166    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"
Sep 12 00:34:12.184619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:12.184203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd" event={"ID":"71fb79d0-60e6-47f3-8266-b1ca5c8b40a6","Type":"ContainerDied","Data":"63445e827955ddbd958183e4fc98e910ddbbe805f030d8e112e1fc6cdd58bb82"}
Sep 12 00:34:12.184619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:12.184250    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63445e827955ddbd958183e4fc98e910ddbbe805f030d8e112e1fc6cdd58bb82"
Sep 12 00:34:12.222212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:12.222176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:12.814996 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:12.814959    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:14.915446 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:14.915407    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:34:18.459402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:18.459359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.007798 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.007747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.109858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.109802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.201212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.201174    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"]
Sep 12 00:34:19.203756 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.203715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.204093 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.204073    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b1c80d8-l6sqd"]
Sep 12 00:34:19.260014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.259934    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.364112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.364082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.570555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.570523    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71fb79d0-60e6-47f3-8266-b1ca5c8b40a6" path="/var/lib/kubelet/pods/71fb79d0-60e6-47f3-8266-b1ca5c8b40a6/volumes"
Sep 12 00:34:19.646868 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.646830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.879934 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.879855    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:19.979905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:19.979868    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:20.105047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.105017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:20.457987 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.457952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:20.524628 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.524592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:20.524820 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.524671    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:20.537390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.537367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:20.558789 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.558755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:20.658737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:20.658706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:21.198350 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:21.198312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:21.243390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:21.243354    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:34:21.956423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:21.956386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:22.743070 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:22.743035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:23.537981 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:23.537950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:23.640256 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:23.640221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:24.428227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:24.428193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:24.528096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:24.528058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:24.823449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:24.823420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:24.924257 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:24.924227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:25.031338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:25.031302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:25.348176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:25.348127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:26.148349 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:26.148311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:26.798872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:26.798836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:27.541389 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:27.541352    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:28.187373 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:28.187335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:28.949813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:28.949774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:30.799382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.799292    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"]
Sep 12 00:34:30.799925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.799904    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71fb79d0-60e6-47f3-8266-b1ca5c8b40a6" containerName="revoke-keys"
Sep 12 00:34:30.799998 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.799927    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71fb79d0-60e6-47f3-8266-b1ca5c8b40a6" containerName="revoke-keys"
Sep 12 00:34:30.800076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.800063    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71fb79d0-60e6-47f3-8266-b1ca5c8b40a6" containerName="revoke-keys"
Sep 12 00:34:30.806655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.806632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:30.809948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.809924    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-139288\"/\"default-dockercfg-sh6fv\""
Sep 12 00:34:30.810174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.810085    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-139288\"/\"kube-root-ca.crt\""
Sep 12 00:34:30.810174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.810093    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-139288\"/\"simulator-w4-139288-kserve-self-signed-certs\""
Sep 12 00:34:30.810174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.810122    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-139288\"/\"openshift-service-ca.crt\""
Sep 12 00:34:30.811038 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.811017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"]
Sep 12 00:34:30.917619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.917578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf4a89-1132-4203-ab1f-873ae5733c25-tls-certs\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:30.917803 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.917634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-dshm\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:30.917803 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.917706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-home\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:30.917803 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.917734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-tmp-dir\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:30.917910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.917804    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rhz5\" (UniqueName: \"kubernetes.io/projected/4fcf4a89-1132-4203-ab1f-873ae5733c25-kube-api-access-8rhz5\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:30.917910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:30.917843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-model-cache\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.007746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.007717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:31.018933 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.018903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-home\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019048 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.018943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-tmp-dir\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019048 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.018968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rhz5\" (UniqueName: \"kubernetes.io/projected/4fcf4a89-1132-4203-ab1f-873ae5733c25-kube-api-access-8rhz5\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019140 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.019117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-model-cache\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.019257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf4a89-1132-4203-ab1f-873ae5733c25-tls-certs\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.019324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-dshm\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.019338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-home\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019461 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.019378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-tmp-dir\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.019461 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.019447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-model-cache\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.021380 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.021352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-dshm\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.021626 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.021610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf4a89-1132-4203-ab1f-873ae5733c25-tls-certs\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.026442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.026421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rhz5\" (UniqueName: \"kubernetes.io/projected/4fcf4a89-1132-4203-ab1f-873ae5733c25-kube-api-access-8rhz5\") pod \"simulator-w4-139288-kserve-88d578f88-5bwtx\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") " pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.117578 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.117542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:31.389839 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:31.389814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"]
Sep 12 00:34:32.296976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:32.296940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx" event={"ID":"4fcf4a89-1132-4203-ab1f-873ae5733c25","Type":"ContainerStarted","Data":"3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873"}
Sep 12 00:34:32.296976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:32.296981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx" event={"ID":"4fcf4a89-1132-4203-ab1f-873ae5733c25","Type":"ContainerStarted","Data":"fbf771b70cb1e559bf8643c5da32ece410124f106b28c6aea72e4666cdf27fbd"}
Sep 12 00:34:32.311992 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:32.311939    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx" podStartSLOduration=2.311923687 podStartE2EDuration="2.311923687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:34:32.310985489 +0000 UTC m=+1075.248292021" watchObservedRunningTime="2026-09-12 00:34:32.311923687 +0000 UTC m=+1075.249230207"
Sep 12 00:34:33.110344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:33.110308    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:34:41.118419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:41.118385    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:41.118419 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:41.118427    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:41.131061 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:41.131038    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:41.356101 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:41.356063    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:34:46.421208 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:46.421176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:47.056794 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:47.056753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:47.773759 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:47.773720    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:48.534317 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.534282    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"]
Sep 12 00:34:48.542542 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.542518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:48.544757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.544732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"]
Sep 12 00:34:48.545181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.545159    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:34:48.550809 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.550767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:48.587054 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.587023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76\" (UID: \"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:48.687795 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.687756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76\" (UID: \"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:48.688412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.688390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76\" (UID: \"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:48.855808 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:48.855770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:49.123279 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:49.123191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"]
Sep 12 00:34:49.126564 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:34:49.126520    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85f59d7_32ee_4a0b_8f09_f1fd28bd35d3.slice/crio-102dafaa5dddeb350e3503dbab159e763217305d292fcf8890e3fa3fbf366ff5 WatchSource:0}: Error finding container 102dafaa5dddeb350e3503dbab159e763217305d292fcf8890e3fa3fbf366ff5: Status 404 returned error can't find the container with id 102dafaa5dddeb350e3503dbab159e763217305d292fcf8890e3fa3fbf366ff5
Sep 12 00:34:49.337277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:49.337246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:49.390221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:49.390136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76" event={"ID":"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3","Type":"ContainerStarted","Data":"102dafaa5dddeb350e3503dbab159e763217305d292fcf8890e3fa3fbf366ff5"}
Sep 12 00:34:50.001726 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:50.001683    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:34:50.050884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:50.050852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:50.396637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:50.396604    2562 generic.go:356] "Generic (PLEG): container finished" podID="e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3" containerID="b62b8acd42c5141fa0f5ba4d37697edd92bfbf83a5aedd9e40b79e1d27c7e1e1" exitCode=0
Sep 12 00:34:50.396818 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:50.396700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76" event={"ID":"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3","Type":"ContainerDied","Data":"b62b8acd42c5141fa0f5ba4d37697edd92bfbf83a5aedd9e40b79e1d27c7e1e1"}
Sep 12 00:34:50.755326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:50.755238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:51.529400 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:51.529363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:51.535800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:51.535775    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:51.616210 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:51.616167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3-maas-api-service-ca\") pod \"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3\" (UID: \"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3\") "
Sep 12 00:34:51.616617 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:51.616595    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3" (UID: "e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:34:51.717493 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:51.717387    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:34:52.182143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:52.182106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:52.409026 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:52.408994    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"
Sep 12 00:34:52.409207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:52.408998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76" event={"ID":"e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3","Type":"ContainerDied","Data":"102dafaa5dddeb350e3503dbab159e763217305d292fcf8890e3fa3fbf366ff5"}
Sep 12 00:34:52.409207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:52.409121    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="102dafaa5dddeb350e3503dbab159e763217305d292fcf8890e3fa3fbf366ff5"
Sep 12 00:34:52.820953 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:52.820916    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:53.526973 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:53.526939    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:54.644689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:54.644650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:54.739903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:54.739873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:55.215974 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.215940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:55.279958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.279904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:55.379593 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.379562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:55.396287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.396258    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"]
Sep 12 00:34:55.398175 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.398155    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6d0c6f2-fdc76"]
Sep 12 00:34:55.479582 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.479496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:55.554375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.554342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:55.577861 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.577820    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3" path="/var/lib/kubelet/pods/e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3/volumes"
Sep 12 00:34:55.835382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:55.835341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:56.017735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:56.017704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:56.356607 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:56.356573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:56.590903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:56.590865    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:56.812939 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:56.812848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:57.356288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:57.356248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:58.198583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:58.198552    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:58.755972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:58.755926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:58.856034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:58.856001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:59.069648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:59.069612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:34:59.845870 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:34:59.845838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:00.711953 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:00.711920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:00.804916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:00.804876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:00.883971 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:00.883923    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:01.168112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:01.168069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:01.261406 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:01.261371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:01.548026 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:01.547944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:02.231967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:02.231888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:02.890197 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:02.890160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:03.544724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:03.544693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:05.305247 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:05.305208    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:35:07.581138 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:07.581105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:22.620909 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:22.620869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:23.263835 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:23.263798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:23.937736 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:23.937690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:24.472462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.472425    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"]
Sep 12 00:35:24.472898 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.472884    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3" containerName="revoke-keys"
Sep 12 00:35:24.472944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.472899    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3" containerName="revoke-keys"
Sep 12 00:35:24.472984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.472979    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85f59d7-32ee-4a0b-8f09-f1fd28bd35d3" containerName="revoke-keys"
Sep 12 00:35:24.476362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.476345    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:24.478749 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.478725    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:35:24.483192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.483169    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"]
Sep 12 00:35:24.641744 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.641697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ad32-bad8-41e6-bce8-7a967e6d1c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh\" (UID: \"d072ad32-bad8-41e6-bce8-7a967e6d1c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:24.743229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.743144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ad32-bad8-41e6-bce8-7a967e6d1c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh\" (UID: \"d072ad32-bad8-41e6-bce8-7a967e6d1c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:24.743889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.743862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ad32-bad8-41e6-bce8-7a967e6d1c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh\" (UID: \"d072ad32-bad8-41e6-bce8-7a967e6d1c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:24.788141 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.788105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:24.929339 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:24.929300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:25.061261 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.061225    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"]
Sep 12 00:35:25.063448 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:35:25.063422    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd072ad32_bad8_41e6_bce8_7a967e6d1c4c.slice/crio-dfb78d05ab41513486ba9257eed5f21fbd0e8554fd3257a6ab54eddb54b8b406 WatchSource:0}: Error finding container dfb78d05ab41513486ba9257eed5f21fbd0e8554fd3257a6ab54eddb54b8b406: Status 404 returned error can't find the container with id dfb78d05ab41513486ba9257eed5f21fbd0e8554fd3257a6ab54eddb54b8b406
Sep 12 00:35:25.590708 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.590667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh" event={"ID":"d072ad32-bad8-41e6-bce8-7a967e6d1c4c","Type":"ContainerStarted","Data":"46a713687fce6ab9e3469e6314e670fe101840ab0a01ee83c0af1f69d6ae254c"}
Sep 12 00:35:25.590878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.590735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh" event={"ID":"d072ad32-bad8-41e6-bce8-7a967e6d1c4c","Type":"ContainerStarted","Data":"dfb78d05ab41513486ba9257eed5f21fbd0e8554fd3257a6ab54eddb54b8b406"}
Sep 12 00:35:25.611881 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.611792    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh" podStartSLOduration=1.611777893 podStartE2EDuration="1.611777893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:35:25.610144987 +0000 UTC m=+1128.547451508" watchObservedRunningTime="2026-09-12 00:35:25.611777893 +0000 UTC m=+1128.549084414"
Sep 12 00:35:25.653650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.653614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:25.918980 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.918897    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"]
Sep 12 00:35:25.922364 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.922340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:25.928843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.928808    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"]
Sep 12 00:35:25.945988 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:25.945950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:26.054649 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.054609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac14e-d820-4daa-9736-98efb1268c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn\" (UID: \"21aac14e-d820-4daa-9736-98efb1268c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:26.155843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.155784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac14e-d820-4daa-9736-98efb1268c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn\" (UID: \"21aac14e-d820-4daa-9736-98efb1268c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:26.156448 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.156425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac14e-d820-4daa-9736-98efb1268c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn\" (UID: \"21aac14e-d820-4daa-9736-98efb1268c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:26.235507 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.235401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:26.509283 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.509199    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"]
Sep 12 00:35:26.512761 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:35:26.512735    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aac14e_d820_4daa_9736_98efb1268c1c.slice/crio-049652975ae22c27e4422dc3e9eac6b9dfd64bce10ceaa2f9ff95f78761a97d1 WatchSource:0}: Error finding container 049652975ae22c27e4422dc3e9eac6b9dfd64bce10ceaa2f9ff95f78761a97d1: Status 404 returned error can't find the container with id 049652975ae22c27e4422dc3e9eac6b9dfd64bce10ceaa2f9ff95f78761a97d1
Sep 12 00:35:26.597595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.597558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn" event={"ID":"21aac14e-d820-4daa-9736-98efb1268c1c","Type":"ContainerStarted","Data":"049652975ae22c27e4422dc3e9eac6b9dfd64bce10ceaa2f9ff95f78761a97d1"}
Sep 12 00:35:26.599121 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.599087    2562 generic.go:356] "Generic (PLEG): container finished" podID="d072ad32-bad8-41e6-bce8-7a967e6d1c4c" containerID="46a713687fce6ab9e3469e6314e670fe101840ab0a01ee83c0af1f69d6ae254c" exitCode=0
Sep 12 00:35:26.599255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.599171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh" event={"ID":"d072ad32-bad8-41e6-bce8-7a967e6d1c4c","Type":"ContainerDied","Data":"46a713687fce6ab9e3469e6314e670fe101840ab0a01ee83c0af1f69d6ae254c"}
Sep 12 00:35:26.604600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:26.604576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:27.451732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.451698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:27.606642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.606608    2562 generic.go:356] "Generic (PLEG): container finished" podID="21aac14e-d820-4daa-9736-98efb1268c1c" containerID="cbb7bb1f72b2b25d5222a46304b31b4c2694716191517c7305ba17abc75fb02e" exitCode=0
Sep 12 00:35:27.606817 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.606678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn" event={"ID":"21aac14e-d820-4daa-9736-98efb1268c1c","Type":"ContainerDied","Data":"cbb7bb1f72b2b25d5222a46304b31b4c2694716191517c7305ba17abc75fb02e"}
Sep 12 00:35:27.746221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.746192    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:27.871394 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.871296    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ad32-bad8-41e6-bce8-7a967e6d1c4c-maas-api-service-ca\") pod \"d072ad32-bad8-41e6-bce8-7a967e6d1c4c\" (UID: \"d072ad32-bad8-41e6-bce8-7a967e6d1c4c\") "
Sep 12 00:35:27.871721 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.871697    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d072ad32-bad8-41e6-bce8-7a967e6d1c4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d072ad32-bad8-41e6-bce8-7a967e6d1c4c" (UID: "d072ad32-bad8-41e6-bce8-7a967e6d1c4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:35:27.972964 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:27.972927    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ad32-bad8-41e6-bce8-7a967e6d1c4c-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:35:28.186787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.186685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:28.613228 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.613183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh" event={"ID":"d072ad32-bad8-41e6-bce8-7a967e6d1c4c","Type":"ContainerDied","Data":"dfb78d05ab41513486ba9257eed5f21fbd0e8554fd3257a6ab54eddb54b8b406"}
Sep 12 00:35:28.613628 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.613244    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb78d05ab41513486ba9257eed5f21fbd0e8554fd3257a6ab54eddb54b8b406"
Sep 12 00:35:28.613628 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.613199    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"
Sep 12 00:35:28.747872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.747845    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:28.870748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.870663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:28.882876 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.882846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac14e-d820-4daa-9736-98efb1268c1c-maas-api-service-ca\") pod \"21aac14e-d820-4daa-9736-98efb1268c1c\" (UID: \"21aac14e-d820-4daa-9736-98efb1268c1c\") "
Sep 12 00:35:28.883203 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.883175    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21aac14e-d820-4daa-9736-98efb1268c1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21aac14e-d820-4daa-9736-98efb1268c1c" (UID: "21aac14e-d820-4daa-9736-98efb1268c1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:35:28.983978 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:28.983943    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac14e-d820-4daa-9736-98efb1268c1c-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:35:29.620722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:29.620692    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"
Sep 12 00:35:29.621110 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:29.620724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn" event={"ID":"21aac14e-d820-4daa-9736-98efb1268c1c","Type":"ContainerDied","Data":"049652975ae22c27e4422dc3e9eac6b9dfd64bce10ceaa2f9ff95f78761a97d1"}
Sep 12 00:35:29.621110 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:29.620772    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="049652975ae22c27e4422dc3e9eac6b9dfd64bce10ceaa2f9ff95f78761a97d1"
Sep 12 00:35:29.648757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:29.648720    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:30.322004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:30.321969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:31.153665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:31.153628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:31.765294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:31.765263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:31.850568 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:31.850532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:31.911814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:31.911775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:32.004928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:32.004884    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"]
Sep 12 00:35:32.007490 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:32.007421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:32.007624 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:32.007561    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cbdb29e-7mpmh"]
Sep 12 00:35:32.067878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:32.067851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:32.214843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:32.214802    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:35:32.523966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:32.523875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:33.221633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:33.221601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:33.351682 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:33.351646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:33.571174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:33.571141    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d072ad32-bad8-41e6-bce8-7a967e6d1c4c" path="/var/lib/kubelet/pods/d072ad32-bad8-41e6-bce8-7a967e6d1c4c/volumes"
Sep 12 00:35:33.683634 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:33.683599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:34.032547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:34.032440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:34.130602 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:34.130566    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:34.551343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:34.551309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:34.911336 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:34.911298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:35.651550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:35.651463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:36.413292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:36.413254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:36.663513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:36.663403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:36.754565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:36.754534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.045214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.045137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.069272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.069237    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"]
Sep 12 00:35:37.069525 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.069493    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx" podUID="4bdea681-0219-4b4e-8c7f-3cc8f4d18520" containerName="payload-processing" containerID="cri-o://87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921" gracePeriod=30
Sep 12 00:35:37.088614 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.088587    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"]
Sep 12 00:35:37.088909 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.088879    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf" podUID="778bd95f-4b67-4c4a-84ce-4a9769598bee" containerName="payload-pre-processing" containerID="cri-o://b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8" gracePeriod=30
Sep 12 00:35:37.114240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.114210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.216674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.216634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.317847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.317813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.338150 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.338126    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:35:37.359276 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.358747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-plugins-config-volume\") pod \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") "
Sep 12 00:35:37.359276 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.358817    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hplnv\" (UniqueName: \"kubernetes.io/projected/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-kube-api-access-hplnv\") pod \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\" (UID: \"4bdea681-0219-4b4e-8c7f-3cc8f4d18520\") "
Sep 12 00:35:37.360185 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.360150    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4bdea681-0219-4b4e-8c7f-3cc8f4d18520" (UID: "4bdea681-0219-4b4e-8c7f-3cc8f4d18520"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:35:37.360368 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.360348    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:35:37.363229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.363199    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-kube-api-access-hplnv" (OuterVolumeSpecName: "kube-api-access-hplnv") pod "4bdea681-0219-4b4e-8c7f-3cc8f4d18520" (UID: "4bdea681-0219-4b4e-8c7f-3cc8f4d18520"). InnerVolumeSpecName "kube-api-access-hplnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:35:37.371407 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.371377    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:35:37.416809 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.416777    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"]
Sep 12 00:35:37.422045 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.422012    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4aaf1a-j2ckn"]
Sep 12 00:35:37.423462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.423421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.461346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.461316    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p69gq\" (UniqueName: \"kubernetes.io/projected/778bd95f-4b67-4c4a-84ce-4a9769598bee-kube-api-access-p69gq\") pod \"778bd95f-4b67-4c4a-84ce-4a9769598bee\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") "
Sep 12 00:35:37.461570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.461371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778bd95f-4b67-4c4a-84ce-4a9769598bee-plugins-config-volume\") pod \"778bd95f-4b67-4c4a-84ce-4a9769598bee\" (UID: \"778bd95f-4b67-4c4a-84ce-4a9769598bee\") "
Sep 12 00:35:37.461707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.461685    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hplnv\" (UniqueName: \"kubernetes.io/projected/4bdea681-0219-4b4e-8c7f-3cc8f4d18520-kube-api-access-hplnv\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:35:37.461805 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.461784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778bd95f-4b67-4c4a-84ce-4a9769598bee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "778bd95f-4b67-4c4a-84ce-4a9769598bee" (UID: "778bd95f-4b67-4c4a-84ce-4a9769598bee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:35:37.463410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.463387    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778bd95f-4b67-4c4a-84ce-4a9769598bee-kube-api-access-p69gq" (OuterVolumeSpecName: "kube-api-access-p69gq") pod "778bd95f-4b67-4c4a-84ce-4a9769598bee" (UID: "778bd95f-4b67-4c4a-84ce-4a9769598bee"). InnerVolumeSpecName "kube-api-access-p69gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:35:37.524822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.524773    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:37.562235 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.562202    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p69gq\" (UniqueName: \"kubernetes.io/projected/778bd95f-4b67-4c4a-84ce-4a9769598bee-kube-api-access-p69gq\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:35:37.562235 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.562233    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/778bd95f-4b67-4c4a-84ce-4a9769598bee-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:35:37.575397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.575359    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21aac14e-d820-4daa-9736-98efb1268c1c" path="/var/lib/kubelet/pods/21aac14e-d820-4daa-9736-98efb1268c1c/volumes"
Sep 12 00:35:37.666592 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.666559    2562 generic.go:356] "Generic (PLEG): container finished" podID="778bd95f-4b67-4c4a-84ce-4a9769598bee" containerID="b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8" exitCode=0
Sep 12 00:35:37.667058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.666625    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"
Sep 12 00:35:37.667058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.666650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf" event={"ID":"778bd95f-4b67-4c4a-84ce-4a9769598bee","Type":"ContainerDied","Data":"b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8"}
Sep 12 00:35:37.667058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.666716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf" event={"ID":"778bd95f-4b67-4c4a-84ce-4a9769598bee","Type":"ContainerDied","Data":"fb55df784a51ebe21164aee1c8679fbac000ef24f1c5235a101489611ab88b9d"}
Sep 12 00:35:37.667058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.666751    2562 scope.go:117] "RemoveContainer" containerID="b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8"
Sep 12 00:35:37.668274 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.668254    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bdea681-0219-4b4e-8c7f-3cc8f4d18520" containerID="87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921" exitCode=0
Sep 12 00:35:37.668369 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.668304    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx" event={"ID":"4bdea681-0219-4b4e-8c7f-3cc8f4d18520","Type":"ContainerDied","Data":"87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921"}
Sep 12 00:35:37.668369 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.668313    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"
Sep 12 00:35:37.668369 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.668343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx" event={"ID":"4bdea681-0219-4b4e-8c7f-3cc8f4d18520","Type":"ContainerDied","Data":"e9cfba904a70c1f4aee67b119347089ec0b25b7ad417f0313e247ed092d1fa30"}
Sep 12 00:35:37.676548 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.676529    2562 scope.go:117] "RemoveContainer" containerID="b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8"
Sep 12 00:35:37.676797 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:35:37.676779    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8\": container with ID starting with b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8 not found: ID does not exist" containerID="b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8"
Sep 12 00:35:37.676924 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.676809    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8"} err="failed to get container status \"b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8\": rpc error: code = NotFound desc = could not find container \"b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8\": container with ID starting with b9af4a2e2f730bcdadd3abfc994e6ebe9b13ff2f361546b088d063137945e8a8 not found: ID does not exist"
Sep 12 00:35:37.676924 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.676833    2562 scope.go:117] "RemoveContainer" containerID="87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921"
Sep 12 00:35:37.684827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.684808    2562 scope.go:117] "RemoveContainer" containerID="87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921"
Sep 12 00:35:37.685103 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:35:37.685077    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921\": container with ID starting with 87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921 not found: ID does not exist" containerID="87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921"
Sep 12 00:35:37.685166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.685110    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921"} err="failed to get container status \"87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921\": rpc error: code = NotFound desc = could not find container \"87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921\": container with ID starting with 87bac5adc267cbd822340a3aa0b56e259349e2646698b1ae37be52f601e26921 not found: ID does not exist"
Sep 12 00:35:37.686425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.686406    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"]
Sep 12 00:35:37.691685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.691662    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4aaf1a-86d778858-d5xcf"]
Sep 12 00:35:37.700922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.700900    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"]
Sep 12 00:35:37.704513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.704494    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4aaf1a-84f46c6b85-wjjsx"]
Sep 12 00:35:37.803647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:37.803577    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:38.133771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:38.133734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:38.234640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:38.234602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:38.419300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:38.419213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:39.059319 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:39.059288    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:39.570690 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:39.570656    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bdea681-0219-4b4e-8c7f-3cc8f4d18520" path="/var/lib/kubelet/pods/4bdea681-0219-4b4e-8c7f-3cc8f4d18520/volumes"
Sep 12 00:35:39.571053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:39.571040    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778bd95f-4b67-4c4a-84ce-4a9769598bee" path="/var/lib/kubelet/pods/778bd95f-4b67-4c4a-84ce-4a9769598bee/volumes"
Sep 12 00:35:39.677353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:39.677272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:40.328746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:40.328716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:41.024406 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024374    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"]
Sep 12 00:35:41.024806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024791    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778bd95f-4b67-4c4a-84ce-4a9769598bee" containerName="payload-pre-processing"
Sep 12 00:35:41.024855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024809    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="778bd95f-4b67-4c4a-84ce-4a9769598bee" containerName="payload-pre-processing"
Sep 12 00:35:41.024855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024827    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21aac14e-d820-4daa-9736-98efb1268c1c" containerName="revoke-keys"
Sep 12 00:35:41.024855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024833    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21aac14e-d820-4daa-9736-98efb1268c1c" containerName="revoke-keys"
Sep 12 00:35:41.024855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024846    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdea681-0219-4b4e-8c7f-3cc8f4d18520" containerName="payload-processing"
Sep 12 00:35:41.024855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024852    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdea681-0219-4b4e-8c7f-3cc8f4d18520" containerName="payload-processing"
Sep 12 00:35:41.025009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024858    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d072ad32-bad8-41e6-bce8-7a967e6d1c4c" containerName="revoke-keys"
Sep 12 00:35:41.025009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024864    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d072ad32-bad8-41e6-bce8-7a967e6d1c4c" containerName="revoke-keys"
Sep 12 00:35:41.025009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024925    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d072ad32-bad8-41e6-bce8-7a967e6d1c4c" containerName="revoke-keys"
Sep 12 00:35:41.025009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024935    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="778bd95f-4b67-4c4a-84ce-4a9769598bee" containerName="payload-pre-processing"
Sep 12 00:35:41.025009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024942    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="21aac14e-d820-4daa-9736-98efb1268c1c" containerName="revoke-keys"
Sep 12 00:35:41.025009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.024950    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bdea681-0219-4b4e-8c7f-3cc8f4d18520" containerName="payload-processing"
Sep 12 00:35:41.029798 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.029781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.032566 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.032542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9c414efe-serving-cert\""
Sep 12 00:35:41.038541 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.038517    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"]
Sep 12 00:35:41.057427 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.057401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:41.093952 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.093915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bjw\" (UniqueName: \"kubernetes.io/projected/eecc5305-b937-4a8d-9acc-d755f11baac6-kube-api-access-s8bjw\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.094072 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.093969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.094129 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.094062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-metrics-certs\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.094168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.094135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eecc5305-b937-4a8d-9acc-d755f11baac6-openshift-service-ca\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.158967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.158936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:41.194775 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.194738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bjw\" (UniqueName: \"kubernetes.io/projected/eecc5305-b937-4a8d-9acc-d755f11baac6-kube-api-access-s8bjw\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.194941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.194792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.194941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.194838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-metrics-certs\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.194941 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.194902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eecc5305-b937-4a8d-9acc-d755f11baac6-openshift-service-ca\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.195063 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:35:41.194972    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-serving-cert: secret "maas-api-e2e-derive-9c414efe-serving-cert" not found
Sep 12 00:35:41.195063 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:35:41.195046    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls podName:eecc5305-b937-4a8d-9acc-d755f11baac6 nodeName:}" failed. No retries permitted until 2026-09-12 00:35:41.695024725 +0000 UTC m=+1144.632331239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls") pod "maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" (UID: "eecc5305-b937-4a8d-9acc-d755f11baac6") : secret "maas-api-e2e-derive-9c414efe-serving-cert" not found
Sep 12 00:35:41.195673 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.195651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eecc5305-b937-4a8d-9acc-d755f11baac6-openshift-service-ca\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.197338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.197316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-metrics-certs\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.202866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.202843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bjw\" (UniqueName: \"kubernetes.io/projected/eecc5305-b937-4a8d-9acc-d755f11baac6-kube-api-access-s8bjw\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.410925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.410872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:41.509592 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.509550    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:41.608465 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.608425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:41.700550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.700445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.702832 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.702805    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls\") pod \"maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.943872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.943836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:41.959547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:41.959511    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:42.230378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:42.230293    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"]
Sep 12 00:35:42.233603 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:35:42.233574    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecc5305_b937_4a8d_9acc_d755f11baac6.slice/crio-42ced5c4a45245c2630221e6701ef05c5fd7f1bbb26415fd70a251f06e3caad5 WatchSource:0}: Error finding container 42ced5c4a45245c2630221e6701ef05c5fd7f1bbb26415fd70a251f06e3caad5: Status 404 returned error can't find the container with id 42ced5c4a45245c2630221e6701ef05c5fd7f1bbb26415fd70a251f06e3caad5
Sep 12 00:35:42.690195 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:42.690163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:42.698994 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:42.698965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" event={"ID":"eecc5305-b937-4a8d-9acc-d755f11baac6","Type":"ContainerStarted","Data":"42ced5c4a45245c2630221e6701ef05c5fd7f1bbb26415fd70a251f06e3caad5"}
Sep 12 00:35:42.893315 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:42.893259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:43.706194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:43.706157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" event={"ID":"eecc5305-b937-4a8d-9acc-d755f11baac6","Type":"ContainerStarted","Data":"cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b"}
Sep 12 00:35:43.706605 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:43.706237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:43.723659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:43.723609    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" podStartSLOduration=2.159661138 podStartE2EDuration="2.723595143s" totalImagesPullingTime="563.934005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:35:43.721894757 +0000 UTC m=+1146.659201274" watchObservedRunningTime="2026-09-12 00:35:43.723595143 +0000 UTC m=+1146.660901717"
Sep 12 00:35:49.625847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:49.625814    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:35:49.718401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:49.718373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:35:54.602009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:35:54.601966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:36:00.307645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:00.307565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:00.564815 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:00.564770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:00.897879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:00.897756    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:01.019454 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:01.019418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:01.335363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:01.335322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:01.677664 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:01.677572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:02.967181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:02.967144    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:03.956248 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:03.956214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:04.648359 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:04.648323    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:04.749291 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:04.749255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:04.853129 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:04.853099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:05.753786 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:05.753749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:06.434753 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:06.434719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:07.471157 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:07.471115    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:08.166647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.166616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:08.223247 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.223211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:08.662812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.662773    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:08.761354 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.761318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:08.794558 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.794512    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"]
Sep 12 00:36:08.798247 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.798228    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:08.802172 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.802149    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-03e1dc\""
Sep 12 00:36:08.802301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.802195    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-03e1dc-dockercfg-hgttx\""
Sep 12 00:36:08.808169 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.808144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"]
Sep 12 00:36:08.966009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.965917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bmv\" (UniqueName: \"kubernetes.io/projected/af29fee6-4c75-46ac-9470-2a8c05ac5151-kube-api-access-z7bmv\") pod \"payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") " pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:08.966009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.965983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af29fee6-4c75-46ac-9470-2a8c05ac5151-plugins-config-volume\") pod \"payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") " pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:08.994802 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:08.994764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:09.067309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.067275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af29fee6-4c75-46ac-9470-2a8c05ac5151-plugins-config-volume\") pod \"payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") " pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:09.067543 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.067389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bmv\" (UniqueName: \"kubernetes.io/projected/af29fee6-4c75-46ac-9470-2a8c05ac5151-kube-api-access-z7bmv\") pod \"payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") " pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:09.068127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.068102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af29fee6-4c75-46ac-9470-2a8c05ac5151-plugins-config-volume\") pod \"payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") " pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:09.074990 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.074966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bmv\" (UniqueName: \"kubernetes.io/projected/af29fee6-4c75-46ac-9470-2a8c05ac5151-kube-api-access-z7bmv\") pod \"payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") " pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:09.095169 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.095143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:09.110078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.110055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:09.381918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.381887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"]
Sep 12 00:36:09.383914 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:36:09.383880    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf29fee6_4c75_46ac_9470_2a8c05ac5151.slice/crio-dc375d37f1142755dad567ee868700450fcd5a11da3570954600ec406e9b32de WatchSource:0}: Error finding container dc375d37f1142755dad567ee868700450fcd5a11da3570954600ec406e9b32de: Status 404 returned error can't find the container with id dc375d37f1142755dad567ee868700450fcd5a11da3570954600ec406e9b32de
Sep 12 00:36:09.789885 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.789788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:09.851827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.851787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq" event={"ID":"af29fee6-4c75-46ac-9470-2a8c05ac5151","Type":"ContainerStarted","Data":"00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb"}
Sep 12 00:36:09.852032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.851854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq" event={"ID":"af29fee6-4c75-46ac-9470-2a8c05ac5151","Type":"ContainerStarted","Data":"dc375d37f1142755dad567ee868700450fcd5a11da3570954600ec406e9b32de"}
Sep 12 00:36:09.852032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.851887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:09.869565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:09.869518    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq" podStartSLOduration=1.86950181 podStartE2EDuration="1.86950181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:36:09.867588005 +0000 UTC m=+1172.804894525" watchObservedRunningTime="2026-09-12 00:36:09.86950181 +0000 UTC m=+1172.806808340"
Sep 12 00:36:10.556137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:10.556103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:11.295488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:11.295445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:12.167971 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:12.167932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:12.897848 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:12.897810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:17.325704 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:17.325662    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:36:20.860619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:20.860582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:36:21.340398 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:21.340360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:28.156218 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.156167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:28.702563 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.702529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"]
Sep 12 00:36:28.702795 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.702762    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" podUID="eecc5305-b937-4a8d-9acc-d755f11baac6" containerName="maas-api" containerID="cri-o://cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b" gracePeriod=30
Sep 12 00:36:28.712230 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.712198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:28.791909 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.791869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:28.855037 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.854993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:28.914793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.914754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:28.970879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.970762    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:36:28.971407 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.971299    2562 generic.go:356] "Generic (PLEG): container finished" podID="eecc5305-b937-4a8d-9acc-d755f11baac6" containerID="cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b" exitCode=0
Sep 12 00:36:28.971595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.971386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" event={"ID":"eecc5305-b937-4a8d-9acc-d755f11baac6","Type":"ContainerDied","Data":"cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b"}
Sep 12 00:36:28.971595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.971527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj" event={"ID":"eecc5305-b937-4a8d-9acc-d755f11baac6","Type":"ContainerDied","Data":"42ced5c4a45245c2630221e6701ef05c5fd7f1bbb26415fd70a251f06e3caad5"}
Sep 12 00:36:28.971595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.971557    2562 scope.go:117] "RemoveContainer" containerID="cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b"
Sep 12 00:36:28.982306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.982276    2562 scope.go:117] "RemoveContainer" containerID="cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b"
Sep 12 00:36:28.982714 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:36:28.982686    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b\": container with ID starting with cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b not found: ID does not exist" containerID="cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b"
Sep 12 00:36:28.982822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:28.982724    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b"} err="failed to get container status \"cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b\": rpc error: code = NotFound desc = could not find container \"cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b\": container with ID starting with cd5fbf0ab5ef6528a69376470ac1562ec2733ad06ee6834f86290be3f6d5194b not found: ID does not exist"
Sep 12 00:36:29.055221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.055181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-metrics-certs\") pod \"eecc5305-b937-4a8d-9acc-d755f11baac6\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") "
Sep 12 00:36:29.055425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.055270    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8bjw\" (UniqueName: \"kubernetes.io/projected/eecc5305-b937-4a8d-9acc-d755f11baac6-kube-api-access-s8bjw\") pod \"eecc5305-b937-4a8d-9acc-d755f11baac6\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") "
Sep 12 00:36:29.055425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.055327    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls\") pod \"eecc5305-b937-4a8d-9acc-d755f11baac6\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") "
Sep 12 00:36:29.055425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.055361    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eecc5305-b937-4a8d-9acc-d755f11baac6-openshift-service-ca\") pod \"eecc5305-b937-4a8d-9acc-d755f11baac6\" (UID: \"eecc5305-b937-4a8d-9acc-d755f11baac6\") "
Sep 12 00:36:29.056030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.055998    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecc5305-b937-4a8d-9acc-d755f11baac6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eecc5305-b937-4a8d-9acc-d755f11baac6" (UID: "eecc5305-b937-4a8d-9acc-d755f11baac6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:36:29.057773 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.057747    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eecc5305-b937-4a8d-9acc-d755f11baac6" (UID: "eecc5305-b937-4a8d-9acc-d755f11baac6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:36:29.059197 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.059170    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecc5305-b937-4a8d-9acc-d755f11baac6-kube-api-access-s8bjw" (OuterVolumeSpecName: "kube-api-access-s8bjw") pod "eecc5305-b937-4a8d-9acc-d755f11baac6" (UID: "eecc5305-b937-4a8d-9acc-d755f11baac6"). InnerVolumeSpecName "kube-api-access-s8bjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:36:29.059197 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.059183    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eecc5305-b937-4a8d-9acc-d755f11baac6" (UID: "eecc5305-b937-4a8d-9acc-d755f11baac6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:36:29.066153 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.066129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:29.156306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.156269    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eecc5305-b937-4a8d-9acc-d755f11baac6-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:36:29.156306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.156302    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:36:29.156306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.156311    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8bjw\" (UniqueName: \"kubernetes.io/projected/eecc5305-b937-4a8d-9acc-d755f11baac6-kube-api-access-s8bjw\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:36:29.156768 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.156320    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eecc5305-b937-4a8d-9acc-d755f11baac6-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:36:29.336328 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.336294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:29.824302 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.824265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:29.925357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.925305    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:29.978404 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.978375    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"
Sep 12 00:36:29.999251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:29.999221    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"]
Sep 12 00:36:30.002284 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:30.002255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:30.003421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:30.003400    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9c414efe-84fcfdc6d7-pxltj"]
Sep 12 00:36:30.462826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:30.462789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:31.344652 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:31.344618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:31.445635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:31.445601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:31.571463 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:31.571416    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eecc5305-b937-4a8d-9acc-d755f11baac6" path="/var/lib/kubelet/pods/eecc5305-b937-4a8d-9acc-d755f11baac6/volumes"
Sep 12 00:36:31.980818 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:31.980765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:32.809204 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:32.809167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:33.548906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:33.548874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:34.318832 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:34.318794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:34.782290 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:34.782203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:35.171905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:35.171873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:36.986547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:36.986512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:37.055303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.055268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:37.140708 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.140672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:37.234724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.234691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:37.335760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.335730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:37.509162 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.509124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:36:37.537096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.537056    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:36:37.545071 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.545045    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:36:37.713760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:37.713678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:38.848906 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:38.848862    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:36:53.301979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:53.301937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:36:59.850412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:36:59.850369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:03.689394 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:03.689350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:09.613767 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:09.613726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:09.721316 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:09.721281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:09.817344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:09.817311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:11.109050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:11.109014    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:37:15.001281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.001247    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"]
Sep 12 00:37:15.001678 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.001665    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eecc5305-b937-4a8d-9acc-d755f11baac6" containerName="maas-api"
Sep 12 00:37:15.001678 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.001679    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eecc5305-b937-4a8d-9acc-d755f11baac6" containerName="maas-api"
Sep 12 00:37:15.001762 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.001753    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eecc5305-b937-4a8d-9acc-d755f11baac6" containerName="maas-api"
Sep 12 00:37:15.004944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.004921    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.007523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.007458    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-03e1dc\"/\"openshift-service-ca.crt\""
Sep 12 00:37:15.008984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.008962    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-03e1dc\"/\"default-dockercfg-mnd7l\""
Sep 12 00:37:15.009086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.009009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-03e1dc\"/\"kube-root-ca.crt\""
Sep 12 00:37:15.009086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.009035    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-03e1dc\"/\"explicit-model-03e1dc-kserve-self-signed-certs\""
Sep 12 00:37:15.013531 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.013509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"]
Sep 12 00:37:15.028200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.028175    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:15.098417 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.098386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tls-certs\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.098582 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.098430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-model-cache\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.098582 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.098464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-dshm\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.098582 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.098558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrqs\" (UniqueName: \"kubernetes.io/projected/35a7f670-4752-4e7e-815c-ec9e0e426cc8-kube-api-access-2hrqs\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.098697 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.098598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tmp-dir\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.098697 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.098617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-home\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.199859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.199820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-home\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200022 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.199967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tls-certs\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200022 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-model-cache\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-dshm\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrqs\" (UniqueName: \"kubernetes.io/projected/35a7f670-4752-4e7e-815c-ec9e0e426cc8-kube-api-access-2hrqs\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tmp-dir\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-home\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-model-cache\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.200442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.200379    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tmp-dir\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.202199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.202177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-dshm\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.202627 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.202605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tls-certs\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.207461 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.207435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrqs\" (UniqueName: \"kubernetes.io/projected/35a7f670-4752-4e7e-815c-ec9e0e426cc8-kube-api-access-2hrqs\") pod \"explicit-model-03e1dc-kserve-bb57987fc-2s75k\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") " pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.315848 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.315822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:15.584834 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:15.584809    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"]
Sep 12 00:37:15.586737 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:37:15.586710    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35a7f670_4752_4e7e_815c_ec9e0e426cc8.slice/crio-dc49ed278809098226c102a7c88df0a077f93723c918f77f317bc704eb780e09 WatchSource:0}: Error finding container dc49ed278809098226c102a7c88df0a077f93723c918f77f317bc704eb780e09: Status 404 returned error can't find the container with id dc49ed278809098226c102a7c88df0a077f93723c918f77f317bc704eb780e09
Sep 12 00:37:16.245730 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:16.245694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k" event={"ID":"35a7f670-4752-4e7e-815c-ec9e0e426cc8","Type":"ContainerStarted","Data":"6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279"}
Sep 12 00:37:16.246116 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:16.245737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k" event={"ID":"35a7f670-4752-4e7e-815c-ec9e0e426cc8","Type":"ContainerStarted","Data":"dc49ed278809098226c102a7c88df0a077f93723c918f77f317bc704eb780e09"}
Sep 12 00:37:16.263717 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:16.263671    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k" podStartSLOduration=2.263657699 podStartE2EDuration="2.263657699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:37:16.262529628 +0000 UTC m=+1239.199836151" watchObservedRunningTime="2026-09-12 00:37:16.263657699 +0000 UTC m=+1239.200964266"
Sep 12 00:37:19.208793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:19.208734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:25.316392 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:25.316345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:25.317127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:25.316510    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:25.329415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:25.329393    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:25.677311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:25.677219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:26.310791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:26.310755    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:29.219054 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.219013    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"]
Sep 12 00:37:29.223255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.223212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:29.338687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.338656    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"]
Sep 12 00:37:29.342047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.342030    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:29.345144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.344828    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:37:29.346740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.346712    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"]
Sep 12 00:37:29.440938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.440898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dd2c-7625-43cc-8cd3-0fedc9822389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-03e1dc-88p79\" (UID: \"f6e9dd2c-7625-43cc-8cd3-0fedc9822389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:29.542520 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.542394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dd2c-7625-43cc-8cd3-0fedc9822389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-03e1dc-88p79\" (UID: \"f6e9dd2c-7625-43cc-8cd3-0fedc9822389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:29.543030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.543010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dd2c-7625-43cc-8cd3-0fedc9822389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-03e1dc-88p79\" (UID: \"f6e9dd2c-7625-43cc-8cd3-0fedc9822389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:29.657176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.657135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:29.741897 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.741857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:29.928048 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:29.928021    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"]
Sep 12 00:37:29.930179 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:37:29.930145    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e9dd2c_7625_43cc_8cd3_0fedc9822389.slice/crio-14d8fe6739082438c4f2984eb2a55edee3e935d96c7461a13abcace685a35134 WatchSource:0}: Error finding container 14d8fe6739082438c4f2984eb2a55edee3e935d96c7461a13abcace685a35134: Status 404 returned error can't find the container with id 14d8fe6739082438c4f2984eb2a55edee3e935d96c7461a13abcace685a35134
Sep 12 00:37:30.323963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:30.323927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79" event={"ID":"f6e9dd2c-7625-43cc-8cd3-0fedc9822389","Type":"ContainerStarted","Data":"7e489bfe79f84d579658c90e02d1d3982d3244a47d89edc4b6f5927fc595435c"}
Sep 12 00:37:30.324345 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:30.323984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79" event={"ID":"f6e9dd2c-7625-43cc-8cd3-0fedc9822389","Type":"ContainerStarted","Data":"14d8fe6739082438c4f2984eb2a55edee3e935d96c7461a13abcace685a35134"}
Sep 12 00:37:30.340172 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:30.340123    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79" podStartSLOduration=1.340109073 podStartE2EDuration="1.340109073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:37:30.337816524 +0000 UTC m=+1253.275123045" watchObservedRunningTime="2026-09-12 00:37:30.340109073 +0000 UTC m=+1253.277415621"
Sep 12 00:37:30.655984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:30.655949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:31.304325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:31.304289    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:31.330772 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:31.330741    2562 generic.go:356] "Generic (PLEG): container finished" podID="f6e9dd2c-7625-43cc-8cd3-0fedc9822389" containerID="7e489bfe79f84d579658c90e02d1d3982d3244a47d89edc4b6f5927fc595435c" exitCode=0
Sep 12 00:37:31.331112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:31.330830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79" event={"ID":"f6e9dd2c-7625-43cc-8cd3-0fedc9822389","Type":"ContainerDied","Data":"7e489bfe79f84d579658c90e02d1d3982d3244a47d89edc4b6f5927fc595435c"}
Sep 12 00:37:31.859134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:31.859086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:31.963464 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:31.963432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:32.059616 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:32.059579    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:32.159560 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:32.159440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:32.472603 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:32.472575    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:32.572536 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:32.572499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dd2c-7625-43cc-8cd3-0fedc9822389-maas-api-service-ca\") pod \"f6e9dd2c-7625-43cc-8cd3-0fedc9822389\" (UID: \"f6e9dd2c-7625-43cc-8cd3-0fedc9822389\") "
Sep 12 00:37:32.572849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:32.572827    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e9dd2c-7625-43cc-8cd3-0fedc9822389-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6e9dd2c-7625-43cc-8cd3-0fedc9822389" (UID: "f6e9dd2c-7625-43cc-8cd3-0fedc9822389"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:37:32.673438 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:32.673393    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9dd2c-7625-43cc-8cd3-0fedc9822389-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:33.044734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:33.044701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:33.344147 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:33.344105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79" event={"ID":"f6e9dd2c-7625-43cc-8cd3-0fedc9822389","Type":"ContainerDied","Data":"14d8fe6739082438c4f2984eb2a55edee3e935d96c7461a13abcace685a35134"}
Sep 12 00:37:33.344325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:33.344170    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d8fe6739082438c4f2984eb2a55edee3e935d96c7461a13abcace685a35134"
Sep 12 00:37:33.344325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:33.344123    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"
Sep 12 00:37:33.836212 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:33.836176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:34.576314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:34.576276    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:38.424555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.424511    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"]
Sep 12 00:37:38.425180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.425162    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e9dd2c-7625-43cc-8cd3-0fedc9822389" containerName="revoke-keys"
Sep 12 00:37:38.425241 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.425183    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e9dd2c-7625-43cc-8cd3-0fedc9822389" containerName="revoke-keys"
Sep 12 00:37:38.425333 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.425321    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e9dd2c-7625-43cc-8cd3-0fedc9822389" containerName="revoke-keys"
Sep 12 00:37:38.428709 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.428688    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.431215 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.431190    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-139288\"/\"e2e-conflict-a-40370e1e-kserve-self-signed-certs\""
Sep 12 00:37:38.435585 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.435560    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"]
Sep 12 00:37:38.461886 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.461853    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:38.526128 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.526080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-home\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.526128 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.526134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa9959a-7c4e-4163-9f57-66721b190d19-tls-certs\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.526366 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.526164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-model-cache\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.526366 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.526205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-dshm\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.526366 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.526307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-tmp-dir\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.526541 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.526374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cdcw\" (UniqueName: \"kubernetes.io/projected/6aa9959a-7c4e-4163-9f57-66721b190d19-kube-api-access-8cdcw\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cdcw\" (UniqueName: \"kubernetes.io/projected/6aa9959a-7c4e-4163-9f57-66721b190d19-kube-api-access-8cdcw\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-home\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa9959a-7c4e-4163-9f57-66721b190d19-tls-certs\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-model-cache\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627623 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-dshm\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627623 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-tmp-dir\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.627963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-model-cache\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.628063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.627987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-tmp-dir\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.628063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.628024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-home\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.629978 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.629956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-dshm\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.630390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.630364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa9959a-7c4e-4163-9f57-66721b190d19-tls-certs\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.635507 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.635483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cdcw\" (UniqueName: \"kubernetes.io/projected/6aa9959a-7c4e-4163-9f57-66721b190d19-kube-api-access-8cdcw\") pod \"e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") " pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:38.739330 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:38.739239    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:39.011991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.011956    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"]
Sep 12 00:37:39.013585 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:37:39.013556    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aa9959a_7c4e_4163_9f57_66721b190d19.slice/crio-8ce6162ddb0f3789b5cf8b1e481571409d0a4ca444d7ab764295446a852d16c7 WatchSource:0}: Error finding container 8ce6162ddb0f3789b5cf8b1e481571409d0a4ca444d7ab764295446a852d16c7: Status 404 returned error can't find the container with id 8ce6162ddb0f3789b5cf8b1e481571409d0a4ca444d7ab764295446a852d16c7
Sep 12 00:37:39.383178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.383143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9" event={"ID":"6aa9959a-7c4e-4163-9f57-66721b190d19","Type":"ContainerStarted","Data":"550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b"}
Sep 12 00:37:39.383361 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.383189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9" event={"ID":"6aa9959a-7c4e-4163-9f57-66721b190d19","Type":"ContainerStarted","Data":"8ce6162ddb0f3789b5cf8b1e481571409d0a4ca444d7ab764295446a852d16c7"}
Sep 12 00:37:39.400696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.400642    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9" podStartSLOduration=1.400626196 podStartE2EDuration="1.400626196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:37:39.398684231 +0000 UTC m=+1262.335990751" watchObservedRunningTime="2026-09-12 00:37:39.400626196 +0000 UTC m=+1262.337932716"
Sep 12 00:37:39.557728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.557693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:39.800769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.800674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:39.823680 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.823643    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"]
Sep 12 00:37:39.823972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.823939    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq" podUID="af29fee6-4c75-46ac-9470-2a8c05ac5151" containerName="payload-processing" containerID="cri-o://00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb" gracePeriod=30
Sep 12 00:37:39.902160 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.902126    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:39.994426 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.994008    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"]
Sep 12 00:37:39.994840 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.994814    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:39.996055 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:39.996030    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-03e1dc-88p79"]
Sep 12 00:37:40.063206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.063173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:40.077819 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.077795    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:37:40.142370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.142333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af29fee6-4c75-46ac-9470-2a8c05ac5151-plugins-config-volume\") pod \"af29fee6-4c75-46ac-9470-2a8c05ac5151\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") "
Sep 12 00:37:40.142590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.142450    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7bmv\" (UniqueName: \"kubernetes.io/projected/af29fee6-4c75-46ac-9470-2a8c05ac5151-kube-api-access-z7bmv\") pod \"af29fee6-4c75-46ac-9470-2a8c05ac5151\" (UID: \"af29fee6-4c75-46ac-9470-2a8c05ac5151\") "
Sep 12 00:37:40.142805 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.142776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af29fee6-4c75-46ac-9470-2a8c05ac5151-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af29fee6-4c75-46ac-9470-2a8c05ac5151" (UID: "af29fee6-4c75-46ac-9470-2a8c05ac5151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:37:40.144655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.144624    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af29fee6-4c75-46ac-9470-2a8c05ac5151-kube-api-access-z7bmv" (OuterVolumeSpecName: "kube-api-access-z7bmv") pod "af29fee6-4c75-46ac-9470-2a8c05ac5151" (UID: "af29fee6-4c75-46ac-9470-2a8c05ac5151"). InnerVolumeSpecName "kube-api-access-z7bmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:37:40.244156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.244115    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af29fee6-4c75-46ac-9470-2a8c05ac5151-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:40.244156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.244149    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bmv\" (UniqueName: \"kubernetes.io/projected/af29fee6-4c75-46ac-9470-2a8c05ac5151-kube-api-access-z7bmv\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:40.359637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.359592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:40.389613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.389580    2562 generic.go:356] "Generic (PLEG): container finished" podID="af29fee6-4c75-46ac-9470-2a8c05ac5151" containerID="00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb" exitCode=0
Sep 12 00:37:40.389787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.389646    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"
Sep 12 00:37:40.389787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.389667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq" event={"ID":"af29fee6-4c75-46ac-9470-2a8c05ac5151","Type":"ContainerDied","Data":"00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb"}
Sep 12 00:37:40.389787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.389725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq" event={"ID":"af29fee6-4c75-46ac-9470-2a8c05ac5151","Type":"ContainerDied","Data":"dc375d37f1142755dad567ee868700450fcd5a11da3570954600ec406e9b32de"}
Sep 12 00:37:40.389787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.389752    2562 scope.go:117] "RemoveContainer" containerID="00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb"
Sep 12 00:37:40.399381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.399362    2562 scope.go:117] "RemoveContainer" containerID="00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb"
Sep 12 00:37:40.399687 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:37:40.399667    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb\": container with ID starting with 00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb not found: ID does not exist" containerID="00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb"
Sep 12 00:37:40.399748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.399698    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb"} err="failed to get container status \"00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb\": rpc error: code = NotFound desc = could not find container \"00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb\": container with ID starting with 00fd35d1bf032b889c1d49ff24dfd58657402bc5c0333db37569f802c111e9cb not found: ID does not exist"
Sep 12 00:37:40.418220 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.418184    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"]
Sep 12 00:37:40.420920 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.420891    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-03e1dc-674d4b6597-nndmq"]
Sep 12 00:37:40.461284 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:40.461247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:41.252549 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:41.252508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:41.570647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:41.570616    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af29fee6-4c75-46ac-9470-2a8c05ac5151" path="/var/lib/kubelet/pods/af29fee6-4c75-46ac-9470-2a8c05ac5151/volumes"
Sep 12 00:37:41.570976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:41.570964    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e9dd2c-7625-43cc-8cd3-0fedc9822389" path="/var/lib/kubelet/pods/f6e9dd2c-7625-43cc-8cd3-0fedc9822389/volumes"
Sep 12 00:37:41.937939 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:41.937842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:42.580191 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:42.580155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:42.680601 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:42.680568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:43.298404 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:43.298374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:43.405689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:43.405653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:44.007517 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.007456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:37:44.336936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.336870    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k" podUID="35a7f670-4752-4e7e-815c-ec9e0e426cc8" containerName="main" containerID="cri-o://6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279" gracePeriod=45
Sep 12 00:37:44.600251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.600185    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:44.686250 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686212    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hrqs\" (UniqueName: \"kubernetes.io/projected/35a7f670-4752-4e7e-815c-ec9e0e426cc8-kube-api-access-2hrqs\") pod \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") "
Sep 12 00:37:44.686432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686272    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tls-certs\") pod \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") "
Sep 12 00:37:44.686432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686298    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-dshm\") pod \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") "
Sep 12 00:37:44.686432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686326    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-home\") pod \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") "
Sep 12 00:37:44.686432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686372    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tmp-dir\") pod \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") "
Sep 12 00:37:44.686680 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686510    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-model-cache\") pod \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\" (UID: \"35a7f670-4752-4e7e-815c-ec9e0e426cc8\") "
Sep 12 00:37:44.686741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-home" (OuterVolumeSpecName: "home") pod "35a7f670-4752-4e7e-815c-ec9e0e426cc8" (UID: "35a7f670-4752-4e7e-815c-ec9e0e426cc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:37:44.686741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686712    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35a7f670-4752-4e7e-815c-ec9e0e426cc8" (UID: "35a7f670-4752-4e7e-815c-ec9e0e426cc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:37:44.686852 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "35a7f670-4752-4e7e-815c-ec9e0e426cc8" (UID: "35a7f670-4752-4e7e-815c-ec9e0e426cc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:37:44.687005 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.686983    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:44.687086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.687008    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:44.687086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.687021    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:44.688747 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.688717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-dshm" (OuterVolumeSpecName: "dshm") pod "35a7f670-4752-4e7e-815c-ec9e0e426cc8" (UID: "35a7f670-4752-4e7e-815c-ec9e0e426cc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:37:44.688862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.688787    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a7f670-4752-4e7e-815c-ec9e0e426cc8-kube-api-access-2hrqs" (OuterVolumeSpecName: "kube-api-access-2hrqs") pod "35a7f670-4752-4e7e-815c-ec9e0e426cc8" (UID: "35a7f670-4752-4e7e-815c-ec9e0e426cc8"). InnerVolumeSpecName "kube-api-access-2hrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:37:44.688862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.688787    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35a7f670-4752-4e7e-815c-ec9e0e426cc8" (UID: "35a7f670-4752-4e7e-815c-ec9e0e426cc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:37:44.788627 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.788592    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrqs\" (UniqueName: \"kubernetes.io/projected/35a7f670-4752-4e7e-815c-ec9e0e426cc8-kube-api-access-2hrqs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:44.788627 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.788626    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a7f670-4752-4e7e-815c-ec9e0e426cc8-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:44.788841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:44.788638    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a7f670-4752-4e7e-815c-ec9e0e426cc8-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:37:45.418548 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.418509    2562 generic.go:356] "Generic (PLEG): container finished" podID="35a7f670-4752-4e7e-815c-ec9e0e426cc8" containerID="6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279" exitCode=0
Sep 12 00:37:45.418925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.418559    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"
Sep 12 00:37:45.418925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.418592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k" event={"ID":"35a7f670-4752-4e7e-815c-ec9e0e426cc8","Type":"ContainerDied","Data":"6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279"}
Sep 12 00:37:45.418925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.418633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k" event={"ID":"35a7f670-4752-4e7e-815c-ec9e0e426cc8","Type":"ContainerDied","Data":"dc49ed278809098226c102a7c88df0a077f93723c918f77f317bc704eb780e09"}
Sep 12 00:37:45.418925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.418654    2562 scope.go:117] "RemoveContainer" containerID="6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279"
Sep 12 00:37:45.428916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.428864    2562 scope.go:117] "RemoveContainer" containerID="6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279"
Sep 12 00:37:45.429132 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.429110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:45.429252 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:37:45.429203    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279\": container with ID starting with 6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279 not found: ID does not exist" containerID="6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279"
Sep 12 00:37:45.429322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.429253    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279"} err="failed to get container status \"6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279\": rpc error: code = NotFound desc = could not find container \"6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279\": container with ID starting with 6cbd36d6a652831ea529d1bca5b7320684680d70738a2aff9e1f0f771ad40279 not found: ID does not exist"
Sep 12 00:37:45.441137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.441092    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"]
Sep 12 00:37:45.445554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.445522    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-03e1dc/explicit-model-03e1dc-kserve-bb57987fc-2s75k"]
Sep 12 00:37:45.571556 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:45.571523    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a7f670-4752-4e7e-815c-ec9e0e426cc8" path="/var/lib/kubelet/pods/35a7f670-4752-4e7e-815c-ec9e0e426cc8/volumes"
Sep 12 00:37:46.196791 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:46.196755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:46.297679 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:46.297643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:46.454801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:46.454715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:46.922529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:46.922491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:47.255662 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:47.255557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:47.473424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:47.473393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:48.265283 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:48.265247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:48.364162 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:48.364118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:48.490793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:48.490756    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:48.739626 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:48.739589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:48.739806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:48.739650    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:48.752509 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:48.752482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:49.465756 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:49.465724    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:37:49.541425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:49.541395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:49.595382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:49.595353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:49.663868 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:49.663842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:49.764980 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:49.764903    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:49.867376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:49.867341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:50.499903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:50.499869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:50.599921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:50.599893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:50.700598 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:50.700570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:52.261415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:52.261382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:57.884937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:57.884901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:57.994057 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:57.994020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:37:58.564982 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:37:58.564947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:00.776192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:00.776157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:04.316112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:04.316072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:04.417669 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:04.417631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:04.557310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:04.557267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:04.846972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:04.846942    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:05.023042 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:05.022980    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:06.945969 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:06.945918    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:07.177783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:07.177747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:07.741349 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:07.741314    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:07.841184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:07.841149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:08.443049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:08.443015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:09.244760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:09.244724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:09.873327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:09.873293    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:10.520749 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.520714    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"]
Sep 12 00:38:10.521165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.521150    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a7f670-4752-4e7e-815c-ec9e0e426cc8" containerName="main"
Sep 12 00:38:10.521206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.521167    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a7f670-4752-4e7e-815c-ec9e0e426cc8" containerName="main"
Sep 12 00:38:10.521206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.521184    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af29fee6-4c75-46ac-9470-2a8c05ac5151" containerName="payload-processing"
Sep 12 00:38:10.521206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.521191    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af29fee6-4c75-46ac-9470-2a8c05ac5151" containerName="payload-processing"
Sep 12 00:38:10.521305 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.521269    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a7f670-4752-4e7e-815c-ec9e0e426cc8" containerName="main"
Sep 12 00:38:10.521305 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.521283    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af29fee6-4c75-46ac-9470-2a8c05ac5151" containerName="payload-processing"
Sep 12 00:38:10.524422 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.524404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.527853 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.527825    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d370bb1a-serving-cert\""
Sep 12 00:38:10.543992 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.543954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"]
Sep 12 00:38:10.571051 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.571023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:10.610071 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.610035    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:38:10.635364 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.635329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.635543 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.635437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-metrics-certs\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.635608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.635540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh798\" (UniqueName: \"kubernetes.io/projected/05585f45-ec80-48d9-8c8e-53bef3275866-kube-api-access-wh798\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.635608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.635572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05585f45-ec80-48d9-8c8e-53bef3275866-openshift-service-ca\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.672401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.672356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:10.736149 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.736103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-metrics-certs\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.736338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.736313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh798\" (UniqueName: \"kubernetes.io/projected/05585f45-ec80-48d9-8c8e-53bef3275866-kube-api-access-wh798\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.736413 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.736354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05585f45-ec80-48d9-8c8e-53bef3275866-openshift-service-ca\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.736504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.736427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.736611 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:38:10.736590    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-serving-cert: secret "maas-api-e2e-mt-d370bb1a-serving-cert" not found
Sep 12 00:38:10.736688 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:38:10.736676    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls podName:05585f45-ec80-48d9-8c8e-53bef3275866 nodeName:}" failed. No retries permitted until 2026-09-12 00:38:11.236656057 +0000 UTC m=+1294.173962568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls") pod "maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" (UID: "05585f45-ec80-48d9-8c8e-53bef3275866") : secret "maas-api-e2e-mt-d370bb1a-serving-cert" not found
Sep 12 00:38:10.737204 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.737176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05585f45-ec80-48d9-8c8e-53bef3275866-openshift-service-ca\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.738918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.738893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-metrics-certs\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.746922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.746895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh798\" (UniqueName: \"kubernetes.io/projected/05585f45-ec80-48d9-8c8e-53bef3275866-kube-api-access-wh798\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:10.985926 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:10.985892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:11.131034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.131000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:11.231987 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.231944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:11.241975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.241911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:11.244397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.244374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls\") pod \"maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:11.314246 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.314215    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"]
Sep 12 00:38:11.319077 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.319058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.321441 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.321418    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d370bb1a-dockercfg-n4m2r\""
Sep 12 00:38:11.321572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.321421    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d370bb1a\""
Sep 12 00:38:11.324799 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.324774    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"]
Sep 12 00:38:11.437196 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.437163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:11.443775 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.443748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.443910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.443796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtklm\" (UniqueName: \"kubernetes.io/projected/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-kube-api-access-qtklm\") pod \"payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.544787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.544753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.544948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.544797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtklm\" (UniqueName: \"kubernetes.io/projected/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-kube-api-access-qtklm\") pod \"payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.545555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.545534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.554226 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.554198    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtklm\" (UniqueName: \"kubernetes.io/projected/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-kube-api-access-qtklm\") pod \"payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.631711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.631676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:11.712539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.712507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"]
Sep 12 00:38:11.714018 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:38:11.713988    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05585f45_ec80_48d9_8c8e_53bef3275866.slice/crio-8d920b814d48ea29455d84e8cd473c654df06a2b988b92e2d02f2b22a6f9bf97 WatchSource:0}: Error finding container 8d920b814d48ea29455d84e8cd473c654df06a2b988b92e2d02f2b22a6f9bf97: Status 404 returned error can't find the container with id 8d920b814d48ea29455d84e8cd473c654df06a2b988b92e2d02f2b22a6f9bf97
Sep 12 00:38:11.902365 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:11.902331    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"]
Sep 12 00:38:11.904098 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:38:11.904076    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca08f713_3a8f_4e2e_b24a_71f187e9d9e2.slice/crio-d725cc52a3d67d17ac80fca1e8e490b0085bb27f7c967f5dcb5ac92fffa7dc49 WatchSource:0}: Error finding container d725cc52a3d67d17ac80fca1e8e490b0085bb27f7c967f5dcb5ac92fffa7dc49: Status 404 returned error can't find the container with id d725cc52a3d67d17ac80fca1e8e490b0085bb27f7c967f5dcb5ac92fffa7dc49
Sep 12 00:38:12.586672 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:12.586623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44" event={"ID":"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2","Type":"ContainerStarted","Data":"93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314"}
Sep 12 00:38:12.587079 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:12.586685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44" event={"ID":"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2","Type":"ContainerStarted","Data":"d725cc52a3d67d17ac80fca1e8e490b0085bb27f7c967f5dcb5ac92fffa7dc49"}
Sep 12 00:38:12.587079 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:12.586744    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:12.588067 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:12.588043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" event={"ID":"05585f45-ec80-48d9-8c8e-53bef3275866","Type":"ContainerStarted","Data":"8d920b814d48ea29455d84e8cd473c654df06a2b988b92e2d02f2b22a6f9bf97"}
Sep 12 00:38:12.604000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:12.603946    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44" podStartSLOduration=1.6039296570000001 podStartE2EDuration="1.603929657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:38:12.601882059 +0000 UTC m=+1295.539188580" watchObservedRunningTime="2026-09-12 00:38:12.603929657 +0000 UTC m=+1295.541236176"
Sep 12 00:38:13.595394 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:13.595352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" event={"ID":"05585f45-ec80-48d9-8c8e-53bef3275866","Type":"ContainerStarted","Data":"20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed"}
Sep 12 00:38:13.595827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:13.595449    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:13.613155 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:13.613090    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" podStartSLOduration=2.805262802 podStartE2EDuration="3.613075231s" totalImagesPullingTime="807.812429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:38:13.611056638 +0000 UTC m=+1296.548363159" watchObservedRunningTime="2026-09-12 00:38:13.613075231 +0000 UTC m=+1296.550381755"
Sep 12 00:38:19.606752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:19.606721    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:23.597631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:23.597595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:32.681363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:32.681323    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:32.781708 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:32.781673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:33.359975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:33.359940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:33.460652 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:33.460616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:33.556292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:33.556257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:33.792242 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:33.792161    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:34.396661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:34.396626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:34.541691 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:34.541652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:35.140770 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:35.140717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:35.777144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:35.777084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:36.548624 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:36.548585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:39.398771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:39.398712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:39.455811 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:39.455767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:40.485521 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:40.485466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:41.355790 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:41.355754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:41.991612 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:41.991571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:42.310763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:42.310688    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:38:42.652381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:42.652348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:43.549513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:43.549480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:44.240957 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:44.240928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:44.871714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:44.871681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:45.845903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:45.845858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:49.053696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:49.053661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:49.871603 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:49.871569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:50.416587 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.416545    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"]
Sep 12 00:38:50.417059 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.416808    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" podUID="05585f45-ec80-48d9-8c8e-53bef3275866" containerName="maas-api" containerID="cri-o://20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed" gracePeriod=30
Sep 12 00:38:50.447420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.447394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:50.487339 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.487303    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"]
Sep 12 00:38:50.487687 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.487655    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44" podUID="ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" containerName="payload-pre-processing" containerID="cri-o://93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314" gracePeriod=30
Sep 12 00:38:50.547922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.547892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:50.648041 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.647962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:50.718368 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.718342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:50.744975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.744954    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:50.776207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.776185    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:50.796782 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.796751    2562 generic.go:356] "Generic (PLEG): container finished" podID="ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" containerID="93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314" exitCode=0
Sep 12 00:38:50.796944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.796783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44" event={"ID":"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2","Type":"ContainerDied","Data":"93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314"}
Sep 12 00:38:50.796944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.796817    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"
Sep 12 00:38:50.796944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.796843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44" event={"ID":"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2","Type":"ContainerDied","Data":"d725cc52a3d67d17ac80fca1e8e490b0085bb27f7c967f5dcb5ac92fffa7dc49"}
Sep 12 00:38:50.796944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.796875    2562 scope.go:117] "RemoveContainer" containerID="93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314"
Sep 12 00:38:50.798341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.798326    2562 generic.go:356] "Generic (PLEG): container finished" podID="05585f45-ec80-48d9-8c8e-53bef3275866" containerID="20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed" exitCode=0
Sep 12 00:38:50.798414 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.798374    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"
Sep 12 00:38:50.798414 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.798376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" event={"ID":"05585f45-ec80-48d9-8c8e-53bef3275866","Type":"ContainerDied","Data":"20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed"}
Sep 12 00:38:50.798635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.798446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw" event={"ID":"05585f45-ec80-48d9-8c8e-53bef3275866","Type":"ContainerDied","Data":"8d920b814d48ea29455d84e8cd473c654df06a2b988b92e2d02f2b22a6f9bf97"}
Sep 12 00:38:50.807879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.807862    2562 scope.go:117] "RemoveContainer" containerID="93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314"
Sep 12 00:38:50.808172 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:38:50.808153    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314\": container with ID starting with 93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314 not found: ID does not exist" containerID="93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314"
Sep 12 00:38:50.808259 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.808184    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314"} err="failed to get container status \"93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314\": rpc error: code = NotFound desc = could not find container \"93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314\": container with ID starting with 93a57d1e6b40bb9811bb5eb609f1a8d70792b2ece7bf8a7694c666a9b5059314 not found: ID does not exist"
Sep 12 00:38:50.808259 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.808206    2562 scope.go:117] "RemoveContainer" containerID="20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed"
Sep 12 00:38:50.817567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.817550    2562 scope.go:117] "RemoveContainer" containerID="20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed"
Sep 12 00:38:50.817857 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:38:50.817838    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed\": container with ID starting with 20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed not found: ID does not exist" containerID="20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed"
Sep 12 00:38:50.817927 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.817866    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed"} err="failed to get container status \"20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed\": rpc error: code = NotFound desc = could not find container \"20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed\": container with ID starting with 20595bee32ca5b2eefe7d3d39e8e2c2d8626a9da8418f058e277283c52e322ed not found: ID does not exist"
Sep 12 00:38:50.822646 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.822627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-metrics-certs\") pod \"05585f45-ec80-48d9-8c8e-53bef3275866\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") "
Sep 12 00:38:50.822740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.822686    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh798\" (UniqueName: \"kubernetes.io/projected/05585f45-ec80-48d9-8c8e-53bef3275866-kube-api-access-wh798\") pod \"05585f45-ec80-48d9-8c8e-53bef3275866\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") "
Sep 12 00:38:50.822802 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.822756    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-plugins-config-volume\") pod \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") "
Sep 12 00:38:50.822802 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.822791    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05585f45-ec80-48d9-8c8e-53bef3275866-openshift-service-ca\") pod \"05585f45-ec80-48d9-8c8e-53bef3275866\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") "
Sep 12 00:38:50.822908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.822877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls\") pod \"05585f45-ec80-48d9-8c8e-53bef3275866\" (UID: \"05585f45-ec80-48d9-8c8e-53bef3275866\") "
Sep 12 00:38:50.822962 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.822915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtklm\" (UniqueName: \"kubernetes.io/projected/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-kube-api-access-qtklm\") pod \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\" (UID: \"ca08f713-3a8f-4e2e-b24a-71f187e9d9e2\") "
Sep 12 00:38:50.823367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.823201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" (UID: "ca08f713-3a8f-4e2e-b24a-71f187e9d9e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:38:50.823367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.823279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05585f45-ec80-48d9-8c8e-53bef3275866-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05585f45-ec80-48d9-8c8e-53bef3275866" (UID: "05585f45-ec80-48d9-8c8e-53bef3275866"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:38:50.823367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.823327    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:38:50.825218 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.825174    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05585f45-ec80-48d9-8c8e-53bef3275866" (UID: "05585f45-ec80-48d9-8c8e-53bef3275866"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:38:50.825301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.825280    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "05585f45-ec80-48d9-8c8e-53bef3275866" (UID: "05585f45-ec80-48d9-8c8e-53bef3275866"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:38:50.825352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.825297    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05585f45-ec80-48d9-8c8e-53bef3275866-kube-api-access-wh798" (OuterVolumeSpecName: "kube-api-access-wh798") pod "05585f45-ec80-48d9-8c8e-53bef3275866" (UID: "05585f45-ec80-48d9-8c8e-53bef3275866"). InnerVolumeSpecName "kube-api-access-wh798". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:38:50.825644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.825621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-kube-api-access-qtklm" (OuterVolumeSpecName: "kube-api-access-qtklm") pod "ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" (UID: "ca08f713-3a8f-4e2e-b24a-71f187e9d9e2"). InnerVolumeSpecName "kube-api-access-qtklm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:38:50.924075 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.924048    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05585f45-ec80-48d9-8c8e-53bef3275866-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:38:50.924075 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.924071    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:38:50.924075 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.924081    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtklm\" (UniqueName: \"kubernetes.io/projected/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2-kube-api-access-qtklm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:38:50.924264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.924089    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05585f45-ec80-48d9-8c8e-53bef3275866-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:38:50.924264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:50.924098    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh798\" (UniqueName: \"kubernetes.io/projected/05585f45-ec80-48d9-8c8e-53bef3275866-kube-api-access-wh798\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:38:51.052809 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.052701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:51.123435 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.123406    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"]
Sep 12 00:38:51.133903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.127331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:51.133903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.130388    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d370bb1a-86567b4f4-x65rw"]
Sep 12 00:38:51.139718 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.139686    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"]
Sep 12 00:38:51.143682 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.143661    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d370bb1a-59fff5bb86-xcc44"]
Sep 12 00:38:51.570743 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.570713    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05585f45-ec80-48d9-8c8e-53bef3275866" path="/var/lib/kubelet/pods/05585f45-ec80-48d9-8c8e-53bef3275866/volumes"
Sep 12 00:38:51.571109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.571087    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" path="/var/lib/kubelet/pods/ca08f713-3a8f-4e2e-b24a-71f187e9d9e2/volumes"
Sep 12 00:38:51.767083 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:51.766993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:52.423030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:52.422994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:53.244748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:53.244709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:53.947029 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:53.946988    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:56.688399 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:56.688367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:56.741629 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:56.741600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:56.943168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:56.943089    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:57.287164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:57.287092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:57.382286 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:57.382257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:38:57.517807 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:38:57.517774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:03.180623 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:03.180527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:03.240661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:03.240629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:03.321153 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:03.321119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:03.424344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:03.424315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:03.532177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:03.532075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:04.810778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:04.810750    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:39:07.646565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:07.646525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:09.290716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:09.290666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:09.350831 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:09.350804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:10.233801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:10.233762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:10.555631 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:10.555544    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:10.741801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:10.741762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:11.229269 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:11.229236    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:12.659102 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:12.659064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:13.313089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:13.313053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:13.463239 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:13.463207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:13.554550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:13.554519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:14.263724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:14.263693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:15.013485 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:15.013439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:15.646633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:15.646595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:16.377807 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.377769    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"]
Sep 12 00:39:16.378251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.378234    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" containerName="payload-pre-processing"
Sep 12 00:39:16.378300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.378253    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" containerName="payload-pre-processing"
Sep 12 00:39:16.378300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.378265    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05585f45-ec80-48d9-8c8e-53bef3275866" containerName="maas-api"
Sep 12 00:39:16.378300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.378271    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05585f45-ec80-48d9-8c8e-53bef3275866" containerName="maas-api"
Sep 12 00:39:16.378400 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.378355    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05585f45-ec80-48d9-8c8e-53bef3275866" containerName="maas-api"
Sep 12 00:39:16.378400 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.378364    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca08f713-3a8f-4e2e-b24a-71f187e9d9e2" containerName="payload-pre-processing"
Sep 12 00:39:16.381424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.381399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.383858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.383831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-0d3a66-serving-cert\""
Sep 12 00:39:16.392699 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.392672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"]
Sep 12 00:39:16.420350 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.420319    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:16.461315 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.461280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/ad029918-7abd-4dee-b03e-a94bbbbae7c7-kube-api-access-49lrj\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.461499 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.461376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad029918-7abd-4dee-b03e-a94bbbbae7c7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.461499 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.461423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.461499 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.461448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.520910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.520879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:16.562135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.562101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.562266 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.562145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.562266 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.562188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/ad029918-7abd-4dee-b03e-a94bbbbae7c7-kube-api-access-49lrj\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.562371 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.562286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad029918-7abd-4dee-b03e-a94bbbbae7c7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.562956 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.562932    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad029918-7abd-4dee-b03e-a94bbbbae7c7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.564869 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.564845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.565000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.564850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.572656 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.572633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/ad029918-7abd-4dee-b03e-a94bbbbae7c7-kube-api-access-49lrj\") pod \"maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.620738 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.620712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:16.693924 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.693858    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:16.963893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.963861    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"]
Sep 12 00:39:16.965863 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:39:16.965833    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad029918_7abd_4dee_b03e_a94bbbbae7c7.slice/crio-40e8f0284abaa749c6bf9ae74ec439b090f74f4a31d010b37ee72c7940393bfc WatchSource:0}: Error finding container 40e8f0284abaa749c6bf9ae74ec439b090f74f4a31d010b37ee72c7940393bfc: Status 404 returned error can't find the container with id 40e8f0284abaa749c6bf9ae74ec439b090f74f4a31d010b37ee72c7940393bfc
Sep 12 00:39:16.967251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.967235    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:39:16.973267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:16.973242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:17.071802 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.071775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:17.080201 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.080171    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"]
Sep 12 00:39:17.085074 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.085058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.088233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.088215    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0d3a66-dockercfg-t9gbt\""
Sep 12 00:39:17.088515 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.088499    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0d3a66\""
Sep 12 00:39:17.096243 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.096223    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"]
Sep 12 00:39:17.168416 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.168381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78890d1-1c85-484e-a5e1-9b43c894c2fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.168594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.168494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ssv\" (UniqueName: \"kubernetes.io/projected/f78890d1-1c85-484e-a5e1-9b43c894c2fe-kube-api-access-55ssv\") pod \"payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.269042 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.268963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ssv\" (UniqueName: \"kubernetes.io/projected/f78890d1-1c85-484e-a5e1-9b43c894c2fe-kube-api-access-55ssv\") pod \"payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.269190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.269046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78890d1-1c85-484e-a5e1-9b43c894c2fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.269721 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.269702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78890d1-1c85-484e-a5e1-9b43c894c2fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.277044 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.277012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ssv\" (UniqueName: \"kubernetes.io/projected/f78890d1-1c85-484e-a5e1-9b43c894c2fe-kube-api-access-55ssv\") pod \"payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.397342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.397314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.672016 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.671987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"]
Sep 12 00:39:17.673744 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:39:17.673717    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78890d1_1c85_484e_a5e1_9b43c894c2fe.slice/crio-8f7247005981c9275369df6fb82c35b529ec69294219c4af1296c2c23d6ae09e WatchSource:0}: Error finding container 8f7247005981c9275369df6fb82c35b529ec69294219c4af1296c2c23d6ae09e: Status 404 returned error can't find the container with id 8f7247005981c9275369df6fb82c35b529ec69294219c4af1296c2c23d6ae09e
Sep 12 00:39:17.949069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.948978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58" event={"ID":"ad029918-7abd-4dee-b03e-a94bbbbae7c7","Type":"ContainerStarted","Data":"3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60"}
Sep 12 00:39:17.949069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.949036    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58" event={"ID":"ad029918-7abd-4dee-b03e-a94bbbbae7c7","Type":"ContainerStarted","Data":"40e8f0284abaa749c6bf9ae74ec439b090f74f4a31d010b37ee72c7940393bfc"}
Sep 12 00:39:17.949597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.949111    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:17.951032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.951004    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6" event={"ID":"f78890d1-1c85-484e-a5e1-9b43c894c2fe","Type":"ContainerStarted","Data":"149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d"}
Sep 12 00:39:17.951180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.951054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6" event={"ID":"f78890d1-1c85-484e-a5e1-9b43c894c2fe","Type":"ContainerStarted","Data":"8f7247005981c9275369df6fb82c35b529ec69294219c4af1296c2c23d6ae09e"}
Sep 12 00:39:17.951180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.951076    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:17.964862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.964812    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58" podStartSLOduration=1.563687379 podStartE2EDuration="1.964796976s" totalImagesPullingTime="401.109597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:17.964426884 +0000 UTC m=+1360.901733407" watchObservedRunningTime="2026-09-12 00:39:17.964796976 +0000 UTC m=+1360.902103541"
Sep 12 00:39:17.979971 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:17.979928    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6" podStartSLOduration=0.979913752 podStartE2EDuration="979.913752ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:17.978040368 +0000 UTC m=+1360.915346888" watchObservedRunningTime="2026-09-12 00:39:17.979913752 +0000 UTC m=+1360.917220273"
Sep 12 00:39:21.560103 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:21.560064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:22.336415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:22.336375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:23.000071 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:23.000034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:23.627939 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:23.627902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:23.964853 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:23.964772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:39:24.453805 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:24.453766    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:25.109440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:25.109396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:25.728235 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:25.728197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:28.960000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:28.959970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:39:30.203482 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:30.203447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:39:32.044570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.044537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:32.073613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.073436    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"]
Sep 12 00:39:32.073792 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.073713    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp" podUID="285c0202-812f-44c3-bff5-5f091280ea0f" containerName="payload-processing" containerID="cri-o://769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b" gracePeriod=30
Sep 12 00:39:32.108741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.108713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:32.168771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.168740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:32.267014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.266982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:32.329644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.329617    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:39:32.349767 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.349733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:32.514156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.514121    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285c0202-812f-44c3-bff5-5f091280ea0f-plugins-config-volume\") pod \"285c0202-812f-44c3-bff5-5f091280ea0f\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") "
Sep 12 00:39:32.514332 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.514178    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk9wg\" (UniqueName: \"kubernetes.io/projected/285c0202-812f-44c3-bff5-5f091280ea0f-kube-api-access-gk9wg\") pod \"285c0202-812f-44c3-bff5-5f091280ea0f\" (UID: \"285c0202-812f-44c3-bff5-5f091280ea0f\") "
Sep 12 00:39:32.514604 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.514578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285c0202-812f-44c3-bff5-5f091280ea0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "285c0202-812f-44c3-bff5-5f091280ea0f" (UID: "285c0202-812f-44c3-bff5-5f091280ea0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:39:32.516267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.516236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285c0202-812f-44c3-bff5-5f091280ea0f-kube-api-access-gk9wg" (OuterVolumeSpecName: "kube-api-access-gk9wg") pod "285c0202-812f-44c3-bff5-5f091280ea0f" (UID: "285c0202-812f-44c3-bff5-5f091280ea0f"). InnerVolumeSpecName "kube-api-access-gk9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:39:32.615976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.615947    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9wg\" (UniqueName: \"kubernetes.io/projected/285c0202-812f-44c3-bff5-5f091280ea0f-kube-api-access-gk9wg\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:39:32.615976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.615974    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285c0202-812f-44c3-bff5-5f091280ea0f-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:39:32.688813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.688784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:32.846274 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:32.846246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:33.036838 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.036754    2562 generic.go:356] "Generic (PLEG): container finished" podID="285c0202-812f-44c3-bff5-5f091280ea0f" containerID="769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b" exitCode=0
Sep 12 00:39:33.036838 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.036800    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"
Sep 12 00:39:33.036838 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.036817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp" event={"ID":"285c0202-812f-44c3-bff5-5f091280ea0f","Type":"ContainerDied","Data":"769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b"}
Sep 12 00:39:33.037047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.036857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp" event={"ID":"285c0202-812f-44c3-bff5-5f091280ea0f","Type":"ContainerDied","Data":"5e0c9923727135e38d8be419d820e72d28dca040de3455ed3bf75ecb01cccbd5"}
Sep 12 00:39:33.037047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.036879    2562 scope.go:117] "RemoveContainer" containerID="769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b"
Sep 12 00:39:33.046619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.046599    2562 scope.go:117] "RemoveContainer" containerID="769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b"
Sep 12 00:39:33.046947 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:39:33.046893    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b\": container with ID starting with 769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b not found: ID does not exist" containerID="769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b"
Sep 12 00:39:33.046947 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.046925    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b"} err="failed to get container status \"769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b\": rpc error: code = NotFound desc = could not find container \"769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b\": container with ID starting with 769c33310c28236ba529d7b133368ec84291a0564535c536fcf52963c51f5a6b not found: ID does not exist"
Sep 12 00:39:33.060134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.060111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"]
Sep 12 00:39:33.062577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.062553    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4df66-869f445c87-cq5cp"]
Sep 12 00:39:33.572135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.572104    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285c0202-812f-44c3-bff5-5f091280ea0f" path="/var/lib/kubelet/pods/285c0202-812f-44c3-bff5-5f091280ea0f/volumes"
Sep 12 00:39:33.576246 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:33.576210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:34.344177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:34.344145    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:35.185064 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:35.185031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:35.872946 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:35.872904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:37.255586 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:37.255548    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:37.736597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:37.736563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:38.060623 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:38.060531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:38.364995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:38.364966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:38.710496 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:38.710391    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:38.810683 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:38.810649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:39.024024 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:39.023937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:39.509249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:39.509219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:39.619440 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:39.619404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:40.275855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:40.275821    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:42.036674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:42.036638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:42.529025 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:42.528992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:42.628362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:42.628327    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:42.728446 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:42.728417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:42.829311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:42.829279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:42.928280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:42.928248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:43.029184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:43.029152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:43.979551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:43.979515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:44.760353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:44.760316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:45.466207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:45.466175    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:46.151268 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.151234    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"]
Sep 12 00:39:46.151671 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.151653    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285c0202-812f-44c3-bff5-5f091280ea0f" containerName="payload-processing"
Sep 12 00:39:46.151671 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.151668    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="285c0202-812f-44c3-bff5-5f091280ea0f" containerName="payload-processing"
Sep 12 00:39:46.151848 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.151780    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="285c0202-812f-44c3-bff5-5f091280ea0f" containerName="payload-processing"
Sep 12 00:39:46.155977 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.155956    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.158507 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.158486    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f12ab7-serving-cert\""
Sep 12 00:39:46.165397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.165367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"]
Sep 12 00:39:46.188763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.188736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:46.238178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.238150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.238282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.238188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.238362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.238339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d189e5-143c-4698-8118-11fcd3daa98e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.238406 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.238387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267r4\" (UniqueName: \"kubernetes.io/projected/a8d189e5-143c-4698-8118-11fcd3daa98e-kube-api-access-267r4\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.242126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.242106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:46.339257 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.339229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.339371 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.339261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.339412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.339370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d189e5-143c-4698-8118-11fcd3daa98e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.339412 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:39:46.339383    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-serving-cert: secret "maas-api-e2e-shared-b-w2-f12ab7-serving-cert" not found
Sep 12 00:39:46.339412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.339405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267r4\" (UniqueName: \"kubernetes.io/projected/a8d189e5-143c-4698-8118-11fcd3daa98e-kube-api-access-267r4\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.339564 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:39:46.339446    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls podName:a8d189e5-143c-4698-8118-11fcd3daa98e nodeName:}" failed. No retries permitted until 2026-09-12 00:39:46.83942744 +0000 UTC m=+1389.776733961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" (UID: "a8d189e5-143c-4698-8118-11fcd3daa98e") : secret "maas-api-e2e-shared-b-w2-f12ab7-serving-cert" not found
Sep 12 00:39:46.340092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.340073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d189e5-143c-4698-8118-11fcd3daa98e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.341708 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.341690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.347374 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.347345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-267r4\" (UniqueName: \"kubernetes.io/projected/a8d189e5-143c-4698-8118-11fcd3daa98e-kube-api-access-267r4\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.467249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.467161    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:46.671383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.671346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:46.764814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.764733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:46.845750 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.845713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:46.848082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:46.848052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:47.070711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:47.070680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:47.342894 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:47.342868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"]
Sep 12 00:39:47.344736 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:39:47.344704    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d189e5_143c_4698_8118_11fcd3daa98e.slice/crio-1b06c47eb3db2b3da9557f9b24f2ce894ed7c95aefe26850e29e834778367d01 WatchSource:0}: Error finding container 1b06c47eb3db2b3da9557f9b24f2ce894ed7c95aefe26850e29e834778367d01: Status 404 returned error can't find the container with id 1b06c47eb3db2b3da9557f9b24f2ce894ed7c95aefe26850e29e834778367d01
Sep 12 00:39:47.897810 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:47.897776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:48.119766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:48.119730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" event={"ID":"a8d189e5-143c-4698-8118-11fcd3daa98e","Type":"ContainerStarted","Data":"fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3"}
Sep 12 00:39:48.119951 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:48.119827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" event={"ID":"a8d189e5-143c-4698-8118-11fcd3daa98e","Type":"ContainerStarted","Data":"1b06c47eb3db2b3da9557f9b24f2ce894ed7c95aefe26850e29e834778367d01"}
Sep 12 00:39:48.119951 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:48.119859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:48.138603 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:48.138555    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" podStartSLOduration=1.834004583 podStartE2EDuration="2.138541436s" totalImagesPullingTime="304.536853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:48.136561177 +0000 UTC m=+1391.073867708" watchObservedRunningTime="2026-09-12 00:39:48.138541436 +0000 UTC m=+1391.075847957"
Sep 12 00:39:52.916213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:52.916177    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:39:54.130868 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:54.130833    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:39:57.983460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:57.983430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:39:58.040832 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:39:58.040783    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:08.377387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:08.377346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:09.262463 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:09.262428    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:09.950055 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:09.950015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:10.757344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:10.757311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:11.493943 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:11.493909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:13.175493 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:13.175443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:13.235691 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:13.235659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:14.406444 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:14.406413    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:14.831323 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:14.831295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:20.114492 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:20.114437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:40:20.382030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:20.381942    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:20.483654 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:20.483618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:20.636435 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:20.636348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:21.164993 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:21.164960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:21.626214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:21.626171    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:21.873032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:21.873000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:21.973348 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:21.973274    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:22.379008 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:22.378976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:22.819281 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:22.819249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:23.488856 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:23.488824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:24.157272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:24.157238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:24.912246 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:24.912213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:25.055917 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:25.055881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:25.192238 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:25.192152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:25.371554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:25.371518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:25.469252 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:25.469161    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:25.683814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:25.683780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:26.476528 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:26.476490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:27.354352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:27.354316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:28.000146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:28.000109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:28.662572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:28.662540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:29.462527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:29.462464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:29.562233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:29.562191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:29.664540 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:29.664503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:29.906805 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:29.906729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:29.995663 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:29.995627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:29.996149 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:29.996127    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"]
Sep 12 00:40:30.000054 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.000038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.002491 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.002449    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cfdf5d6c\""
Sep 12 00:40:30.009108 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.009080    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"]
Sep 12 00:40:30.033138 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.033109    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"]
Sep 12 00:40:30.036781 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.036765    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.039310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.039290    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cfdf5d6c-dockercfg-6b8mr\""
Sep 12 00:40:30.056681 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.056643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hzn\" (UniqueName: \"kubernetes.io/projected/972c3828-67ef-429a-91ea-9722f123acd2-kube-api-access-d4hzn\") pod \"payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.056822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.056803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972c3828-67ef-429a-91ea-9722f123acd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.061379 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.061348    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"]
Sep 12 00:40:30.157728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.157629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972c3828-67ef-429a-91ea-9722f123acd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.157886 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.157762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hzn\" (UniqueName: \"kubernetes.io/projected/972c3828-67ef-429a-91ea-9722f123acd2-kube-api-access-d4hzn\") pod \"payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.157886 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.157865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-plugins-config-volume\") pod \"payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") " pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.157984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.157914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-kube-api-access-pwvzf\") pod \"payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") " pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.158377 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.158355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972c3828-67ef-429a-91ea-9722f123acd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.165336 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.165308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hzn\" (UniqueName: \"kubernetes.io/projected/972c3828-67ef-429a-91ea-9722f123acd2-kube-api-access-d4hzn\") pod \"payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.259411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.259363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-plugins-config-volume\") pod \"payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") " pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.259619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.259421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-kube-api-access-pwvzf\") pod \"payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") " pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.260132 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.260112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-plugins-config-volume\") pod \"payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") " pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.268306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.268275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-kube-api-access-pwvzf\") pod \"payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") " pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.312192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.312151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:30.348849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.348806    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:30.593401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.593371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"]
Sep 12 00:40:30.595150 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:40:30.595123    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972c3828_67ef_429a_91ea_9722f123acd2.slice/crio-a8b39fb032c53763f64c35d3941f04bb6185653d9e89ac5129a3ebe755987874 WatchSource:0}: Error finding container a8b39fb032c53763f64c35d3941f04bb6185653d9e89ac5129a3ebe755987874: Status 404 returned error can't find the container with id a8b39fb032c53763f64c35d3941f04bb6185653d9e89ac5129a3ebe755987874
Sep 12 00:40:30.635360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:30.635333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"]
Sep 12 00:40:30.636785 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:40:30.636760    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd28fc5_bd83_47b9_8606_2723b25b8ba3.slice/crio-bdf153fad3f42bc74934eb3dbf4b3904f7bfa2c497e9b85496ef4220d70bca85 WatchSource:0}: Error finding container bdf153fad3f42bc74934eb3dbf4b3904f7bfa2c497e9b85496ef4220d70bca85: Status 404 returned error can't find the container with id bdf153fad3f42bc74934eb3dbf4b3904f7bfa2c497e9b85496ef4220d70bca85
Sep 12 00:40:31.133273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.132850    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"]
Sep 12 00:40:31.133273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.133087    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" podUID="36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" containerName="maas-api" containerID="cri-o://f1a378cae17c10cf51967bfe6aa0082862526e2fe7f977dd2c470fec4ea2d975" gracePeriod=30
Sep 12 00:40:31.158532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.158496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:31.201094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.201060    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"]
Sep 12 00:40:31.201339 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.201313    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76" podUID="96ede5fa-47c9-4e3b-b2db-e09356a32b65" containerName="payload-pre-processing" containerID="cri-o://330a6de610bea5401f879a0aa2e7299d8a34d671b74c57973548cb45ede14b8d" gracePeriod=30
Sep 12 00:40:31.223607 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.223507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:31.321025 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.320998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:31.365597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.365119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n" event={"ID":"0fd28fc5-bd83-47b9-8606-2723b25b8ba3","Type":"ContainerStarted","Data":"2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10"}
Sep 12 00:40:31.365597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.365172    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n" event={"ID":"0fd28fc5-bd83-47b9-8606-2723b25b8ba3","Type":"ContainerStarted","Data":"bdf153fad3f42bc74934eb3dbf4b3904f7bfa2c497e9b85496ef4220d70bca85"}
Sep 12 00:40:31.365597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.365412    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:31.367367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.367340    2562 generic.go:356] "Generic (PLEG): container finished" podID="96ede5fa-47c9-4e3b-b2db-e09356a32b65" containerID="330a6de610bea5401f879a0aa2e7299d8a34d671b74c57973548cb45ede14b8d" exitCode=0
Sep 12 00:40:31.367541 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.367424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76" event={"ID":"96ede5fa-47c9-4e3b-b2db-e09356a32b65","Type":"ContainerDied","Data":"330a6de610bea5401f879a0aa2e7299d8a34d671b74c57973548cb45ede14b8d"}
Sep 12 00:40:31.369962 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.369937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf" event={"ID":"972c3828-67ef-429a-91ea-9722f123acd2","Type":"ContainerStarted","Data":"8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148"}
Sep 12 00:40:31.370080 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.370002    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:31.370080 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.370027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf" event={"ID":"972c3828-67ef-429a-91ea-9722f123acd2","Type":"ContainerStarted","Data":"a8b39fb032c53763f64c35d3941f04bb6185653d9e89ac5129a3ebe755987874"}
Sep 12 00:40:31.372776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.372627    2562 generic.go:356] "Generic (PLEG): container finished" podID="36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" containerID="f1a378cae17c10cf51967bfe6aa0082862526e2fe7f977dd2c470fec4ea2d975" exitCode=0
Sep 12 00:40:31.372776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.372695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" event={"ID":"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65","Type":"ContainerDied","Data":"f1a378cae17c10cf51967bfe6aa0082862526e2fe7f977dd2c470fec4ea2d975"}
Sep 12 00:40:31.375439 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.375202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:31.381748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.380892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n" podStartSLOduration=1.3808744609999999 podStartE2EDuration="1.380874461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:40:31.380357323 +0000 UTC m=+1434.317663859" watchObservedRunningTime="2026-09-12 00:40:31.380874461 +0000 UTC m=+1434.318180982"
Sep 12 00:40:31.401184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.401075    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf" podStartSLOduration=2.401058321 podStartE2EDuration="2.401058321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:40:31.39886243 +0000 UTC m=+1434.336168947" watchObservedRunningTime="2026-09-12 00:40:31.401058321 +0000 UTC m=+1434.338364833"
Sep 12 00:40:31.465462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.465432    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:40:31.504905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.504878    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:40:31.571892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.571857    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-metrics-certs\") pod \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") "
Sep 12 00:40:31.572103 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.572089    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzdsh\" (UniqueName: \"kubernetes.io/projected/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-kube-api-access-vzdsh\") pod \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") "
Sep 12 00:40:31.572177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.572139    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls\") pod \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") "
Sep 12 00:40:31.572309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.572281    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ede5fa-47c9-4e3b-b2db-e09356a32b65-plugins-config-volume\") pod \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") "
Sep 12 00:40:31.572711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.572676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ede5fa-47c9-4e3b-b2db-e09356a32b65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96ede5fa-47c9-4e3b-b2db-e09356a32b65" (UID: "96ede5fa-47c9-4e3b-b2db-e09356a32b65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:40:31.572805 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.572760    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-openshift-service-ca\") pod \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\" (UID: \"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65\") "
Sep 12 00:40:31.572878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.572804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz9lf\" (UniqueName: \"kubernetes.io/projected/96ede5fa-47c9-4e3b-b2db-e09356a32b65-kube-api-access-zz9lf\") pod \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\" (UID: \"96ede5fa-47c9-4e3b-b2db-e09356a32b65\") "
Sep 12 00:40:31.573270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.573160    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" (UID: "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:40:31.573371 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.573346    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ede5fa-47c9-4e3b-b2db-e09356a32b65-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:40:31.573435 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.573368    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:40:31.574425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.574396    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" (UID: "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:40:31.574648 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.574620    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" (UID: "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:40:31.574729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.574665    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-kube-api-access-vzdsh" (OuterVolumeSpecName: "kube-api-access-vzdsh") pod "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" (UID: "36a847b9-db0b-4d3f-8b1c-d2b2d33aee65"). InnerVolumeSpecName "kube-api-access-vzdsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:40:31.575661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.575644    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ede5fa-47c9-4e3b-b2db-e09356a32b65-kube-api-access-zz9lf" (OuterVolumeSpecName: "kube-api-access-zz9lf") pod "96ede5fa-47c9-4e3b-b2db-e09356a32b65" (UID: "96ede5fa-47c9-4e3b-b2db-e09356a32b65"). InnerVolumeSpecName "kube-api-access-zz9lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:40:31.674372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.674285    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:40:31.674372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.674320    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdsh\" (UniqueName: \"kubernetes.io/projected/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-kube-api-access-vzdsh\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:40:31.674372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.674331    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:40:31.674372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.674340    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz9lf\" (UniqueName: \"kubernetes.io/projected/96ede5fa-47c9-4e3b-b2db-e09356a32b65-kube-api-access-zz9lf\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:40:31.745804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.745757    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:31.846008 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:31.845975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:32.380357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.380314    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"
Sep 12 00:40:32.380357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.380340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq" event={"ID":"36a847b9-db0b-4d3f-8b1c-d2b2d33aee65","Type":"ContainerDied","Data":"6bb5f95e68bd512801320d956c9de2f7b7817b219f23c78e316378b6b4c74bea"}
Sep 12 00:40:32.380633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.380409    2562 scope.go:117] "RemoveContainer" containerID="f1a378cae17c10cf51967bfe6aa0082862526e2fe7f977dd2c470fec4ea2d975"
Sep 12 00:40:32.381894 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.381794    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"
Sep 12 00:40:32.381894 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.381827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76" event={"ID":"96ede5fa-47c9-4e3b-b2db-e09356a32b65","Type":"ContainerDied","Data":"f8bb658126728ea50093d5dbe7e81228f7de8ec1a96e3310a5f9a256d20c1f7d"}
Sep 12 00:40:32.390718 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.390694    2562 scope.go:117] "RemoveContainer" containerID="330a6de610bea5401f879a0aa2e7299d8a34d671b74c57973548cb45ede14b8d"
Sep 12 00:40:32.406135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.406111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"]
Sep 12 00:40:32.410462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.410440    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39d42b-6b8b9865d6-fmd76"]
Sep 12 00:40:32.422739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.422716    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"]
Sep 12 00:40:32.426853 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.426833    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39d42b-d6f4c9b87-8rhrq"]
Sep 12 00:40:32.450561 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:32.450467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:33.221442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:33.221407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:33.570536 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:33.570502    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" path="/var/lib/kubelet/pods/36a847b9-db0b-4d3f-8b1c-d2b2d33aee65/volumes"
Sep 12 00:40:33.570899 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:33.570886    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ede5fa-47c9-4e3b-b2db-e09356a32b65" path="/var/lib/kubelet/pods/96ede5fa-47c9-4e3b-b2db-e09356a32b65/volumes"
Sep 12 00:40:34.138735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:34.138702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:34.784714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:34.784669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:35.428282 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:35.428250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:36.870644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:36.870616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:37.635609 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:37.635584    2562 scope.go:117] "RemoveContainer" containerID="60b412f45d4dca9ed9d8fc441ced5c0615be6f62eba314b5642b1d33c5f0dff6"
Sep 12 00:40:42.384351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:42.384316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:40:42.384808 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:42.384390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:40:47.163110 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:47.163069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:47.213279 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:47.213248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:47.887165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:47.887125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:48.290613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:48.290528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:48.992819 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:48.992784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:49.652097 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:49.652062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:50.014910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:50.014831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:50.366251 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:50.366212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:51.248675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:51.248632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:51.940415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:51.940385    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:52.752218 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:52.752190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:52.885615 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:52.885582    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"]
Sep 12 00:40:52.966736 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:52.966700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:53.379859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:53.379829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:54.037243 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:54.037178    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:54.754069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:54.754039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:55.429258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:55.429216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:55.526041 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:55.526008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:56.601484 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:56.601438    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:40:58.054036 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:58.054001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:58.167161 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:58.167114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:40:58.415132 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:40:58.415095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:00.398795 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.398762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:00.589781 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.589741    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"]
Sep 12 00:41:00.590263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.590243    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ede5fa-47c9-4e3b-b2db-e09356a32b65" containerName="payload-pre-processing"
Sep 12 00:41:00.590263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.590265    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ede5fa-47c9-4e3b-b2db-e09356a32b65" containerName="payload-pre-processing"
Sep 12 00:41:00.590418 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.590324    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" containerName="maas-api"
Sep 12 00:41:00.590418 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.590334    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" containerName="maas-api"
Sep 12 00:41:00.590550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.590431    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ede5fa-47c9-4e3b-b2db-e09356a32b65" containerName="payload-pre-processing"
Sep 12 00:41:00.590550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.590450    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a847b9-db0b-4d3f-8b1c-d2b2d33aee65" containerName="maas-api"
Sep 12 00:41:00.595578 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.595553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.599390 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.599361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cfdf5d6c\"/\"e2e-lifecycle-model-cfdf5d6c-kserve-self-signed-certs\""
Sep 12 00:41:00.600384 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.600367    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cfdf5d6c\"/\"kube-root-ca.crt\""
Sep 12 00:41:00.600504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.600368    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cfdf5d6c\"/\"default-dockercfg-tsfc7\""
Sep 12 00:41:00.600776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.600759    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cfdf5d6c\"/\"openshift-service-ca.crt\""
Sep 12 00:41:00.607572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.607548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"]
Sep 12 00:41:00.642324 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.642290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:00.657690 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.657592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-model-cache\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.657690 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.657632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5ea539-9326-4f63-98f0-57b38d3c8034-tls-certs\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.657690 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.657651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-dshm\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.657938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.657794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grtz\" (UniqueName: \"kubernetes.io/projected/fa5ea539-9326-4f63-98f0-57b38d3c8034-kube-api-access-5grtz\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.657938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.657841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-tmp-dir\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.657938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.657872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-home\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.729308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.729275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:00.759009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.758970    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grtz\" (UniqueName: \"kubernetes.io/projected/fa5ea539-9326-4f63-98f0-57b38d3c8034-kube-api-access-5grtz\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-tmp-dir\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-home\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-model-cache\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5ea539-9326-4f63-98f0-57b38d3c8034-tls-certs\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-dshm\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759487 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-tmp-dir\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759549 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-home\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.759592 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.759573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-model-cache\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.761729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.761696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-dshm\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.761855 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.761806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5ea539-9326-4f63-98f0-57b38d3c8034-tls-certs\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.766697 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.766673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grtz\" (UniqueName: \"kubernetes.io/projected/fa5ea539-9326-4f63-98f0-57b38d3c8034-kube-api-access-5grtz\") pod \"e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") " pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:00.906003 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:00.905958    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:01.182794 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:01.182768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"]
Sep 12 00:41:01.184286 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:41:01.184258    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa5ea539_9326_4f63_98f0_57b38d3c8034.slice/crio-61e057b4b7c370fc8ceda4ea27026dd93c56c9dede1d652fd237b202d1587497 WatchSource:0}: Error finding container 61e057b4b7c370fc8ceda4ea27026dd93c56c9dede1d652fd237b202d1587497: Status 404 returned error can't find the container with id 61e057b4b7c370fc8ceda4ea27026dd93c56c9dede1d652fd237b202d1587497
Sep 12 00:41:01.269774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:01.269724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:01.554029 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:01.553928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs" event={"ID":"fa5ea539-9326-4f63-98f0-57b38d3c8034","Type":"ContainerStarted","Data":"90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4"}
Sep 12 00:41:01.554029 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:01.553979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs" event={"ID":"fa5ea539-9326-4f63-98f0-57b38d3c8034","Type":"ContainerStarted","Data":"61e057b4b7c370fc8ceda4ea27026dd93c56c9dede1d652fd237b202d1587497"}
Sep 12 00:41:01.571981 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:01.571927    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs" podStartSLOduration=1.571910143 podStartE2EDuration="1.571910143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:41:01.570883626 +0000 UTC m=+1464.508190145" watchObservedRunningTime="2026-09-12 00:41:01.571910143 +0000 UTC m=+1464.509216663"
Sep 12 00:41:01.621465 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:01.621418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:02.197830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:02.197798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.004417 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.004381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.071221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.071188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.130801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.130773    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.232031 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.231997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.439817 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.439785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.617268 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.617232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:04.869690 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:04.869651    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:05.040035 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:05.039993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:05.141118 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:05.141021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:05.474549 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:05.474451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:06.269698 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:06.269661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:06.908655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:06.908616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:07.513406 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:07.513372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:07.652331 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:07.652219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:07.905058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:07.905024    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6" podUID="62dc8359-4a2c-44c1-b387-5840a02b1c48" containerName="main" containerID="cri-o://d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987" gracePeriod=45
Sep 12 00:41:08.152622 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.152599    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:41:08.233123 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233037    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-model-cache\") pod \"62dc8359-4a2c-44c1-b387-5840a02b1c48\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") "
Sep 12 00:41:08.233123 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-home\") pod \"62dc8359-4a2c-44c1-b387-5840a02b1c48\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") "
Sep 12 00:41:08.233351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gqws\" (UniqueName: \"kubernetes.io/projected/62dc8359-4a2c-44c1-b387-5840a02b1c48-kube-api-access-2gqws\") pod \"62dc8359-4a2c-44c1-b387-5840a02b1c48\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") "
Sep 12 00:41:08.233351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233177    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-tmp-dir\") pod \"62dc8359-4a2c-44c1-b387-5840a02b1c48\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") "
Sep 12 00:41:08.233351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233270    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc8359-4a2c-44c1-b387-5840a02b1c48-tls-certs\") pod \"62dc8359-4a2c-44c1-b387-5840a02b1c48\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") "
Sep 12 00:41:08.233351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-model-cache" (OuterVolumeSpecName: "model-cache") pod "62dc8359-4a2c-44c1-b387-5840a02b1c48" (UID: "62dc8359-4a2c-44c1-b387-5840a02b1c48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:08.233351 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-dshm\") pod \"62dc8359-4a2c-44c1-b387-5840a02b1c48\" (UID: \"62dc8359-4a2c-44c1-b387-5840a02b1c48\") "
Sep 12 00:41:08.233642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233428    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-home" (OuterVolumeSpecName: "home") pod "62dc8359-4a2c-44c1-b387-5840a02b1c48" (UID: "62dc8359-4a2c-44c1-b387-5840a02b1c48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:08.233642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233516    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62dc8359-4a2c-44c1-b387-5840a02b1c48" (UID: "62dc8359-4a2c-44c1-b387-5840a02b1c48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:08.233765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233744    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:08.233813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233767    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:08.233813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.233779    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:08.235719 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.235688    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dc8359-4a2c-44c1-b387-5840a02b1c48-kube-api-access-2gqws" (OuterVolumeSpecName: "kube-api-access-2gqws") pod "62dc8359-4a2c-44c1-b387-5840a02b1c48" (UID: "62dc8359-4a2c-44c1-b387-5840a02b1c48"). InnerVolumeSpecName "kube-api-access-2gqws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:41:08.235858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.235812    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-dshm" (OuterVolumeSpecName: "dshm") pod "62dc8359-4a2c-44c1-b387-5840a02b1c48" (UID: "62dc8359-4a2c-44c1-b387-5840a02b1c48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:08.235913 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.235868    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62dc8359-4a2c-44c1-b387-5840a02b1c48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62dc8359-4a2c-44c1-b387-5840a02b1c48" (UID: "62dc8359-4a2c-44c1-b387-5840a02b1c48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:41:08.334307 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.334281    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc8359-4a2c-44c1-b387-5840a02b1c48-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:08.334307 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.334305    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc8359-4a2c-44c1-b387-5840a02b1c48-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:08.334507 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.334315    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqws\" (UniqueName: \"kubernetes.io/projected/62dc8359-4a2c-44c1-b387-5840a02b1c48-kube-api-access-2gqws\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:08.361213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.361181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:08.420513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.420455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:08.592645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.592613    2562 generic.go:356] "Generic (PLEG): container finished" podID="62dc8359-4a2c-44c1-b387-5840a02b1c48" containerID="d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987" exitCode=0
Sep 12 00:41:08.593062 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.592657    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"
Sep 12 00:41:08.593062 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.592698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6" event={"ID":"62dc8359-4a2c-44c1-b387-5840a02b1c48","Type":"ContainerDied","Data":"d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987"}
Sep 12 00:41:08.593062 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.592740    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6" event={"ID":"62dc8359-4a2c-44c1-b387-5840a02b1c48","Type":"ContainerDied","Data":"8d50d2b2df491556c286bbf5feaa8d10acab25b1a43f4935c23a700cbf87b1aa"}
Sep 12 00:41:08.593062 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.592759    2562 scope.go:117] "RemoveContainer" containerID="d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987"
Sep 12 00:41:08.609726 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.609679    2562 scope.go:117] "RemoveContainer" containerID="d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987"
Sep 12 00:41:08.610000 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:41:08.609981    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987\": container with ID starting with d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987 not found: ID does not exist" containerID="d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987"
Sep 12 00:41:08.610040 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.610009    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987"} err="failed to get container status \"d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987\": rpc error: code = NotFound desc = could not find container \"d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987\": container with ID starting with d007c599c183987725f0d2e1a6a18c95341be8282efa3ea2883442ea2e025987 not found: ID does not exist"
Sep 12 00:41:08.618605 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.618581    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"]
Sep 12 00:41:08.624714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.624691    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3c466c/simulator-w0-3c466c-kserve-5c59b76644-gzdp6"]
Sep 12 00:41:08.662392 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.662372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:08.828329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.828298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:08.932908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:08.932828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:09.247932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:09.247842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:09.571540 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:09.571509    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dc8359-4a2c-44c1-b387-5840a02b1c48" path="/var/lib/kubelet/pods/62dc8359-4a2c-44c1-b387-5840a02b1c48/volumes"
Sep 12 00:41:10.067257 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:10.067229    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:10.838802 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:10.838747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:10.907014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:10.906983    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:10.907176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:10.907029    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:10.919836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:10.919802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:10.938428 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:10.938362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:11.097639 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:11.097538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:11.619983 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:11.619952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:41:26.225100 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:26.225067    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"]
Sep 12 00:41:26.255613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:26.255555    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:27.003457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:27.003418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:41:29.020568 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:29.020534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:29.125020 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:29.124969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:30.426116 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:30.426058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:30.743077 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:30.742975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:31.380896 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:31.380866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:32.033367 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:32.033334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:32.682048 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:32.682012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:33.456458 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:33.456418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:34.225342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:34.225304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:35.171043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:35.171010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:35.512189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:35.512106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:35.818514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:35.818455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:36.006781 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:36.006745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:37.575844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:37.575818    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:41:37.584900 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:37.584872    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:41:37.668325 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:37.668285    2562 scope.go:117] "RemoveContainer" containerID="46a713687fce6ab9e3469e6314e670fe101840ab0a01ee83c0af1f69d6ae254c"
Sep 12 00:41:37.676925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:37.676904    2562 scope.go:117] "RemoveContainer" containerID="cbb7bb1f72b2b25d5222a46304b31b4c2694716191517c7305ba17abc75fb02e"
Sep 12 00:41:37.684763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:37.684744    2562 scope.go:117] "RemoveContainer" containerID="b62b8acd42c5141fa0f5ba4d37697edd92bfbf83a5aedd9e40b79e1d27c7e1e1"
Sep 12 00:41:38.211619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:38.211581    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:38.843423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:38.843388    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:38.943219 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:38.943174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:39.668811 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:39.668776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:40.464302 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:40.464270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:41.240928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.240890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:41.245457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.245423    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9" podUID="6aa9959a-7c4e-4163-9f57-66721b190d19" containerName="main" containerID="cri-o://550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b" gracePeriod=45
Sep 12 00:41:41.498078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.498021    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:41:41.579360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579332    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cdcw\" (UniqueName: \"kubernetes.io/projected/6aa9959a-7c4e-4163-9f57-66721b190d19-kube-api-access-8cdcw\") pod \"6aa9959a-7c4e-4163-9f57-66721b190d19\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") "
Sep 12 00:41:41.579523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579368    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-dshm\") pod \"6aa9959a-7c4e-4163-9f57-66721b190d19\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") "
Sep 12 00:41:41.579523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-tmp-dir\") pod \"6aa9959a-7c4e-4163-9f57-66721b190d19\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") "
Sep 12 00:41:41.579523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579507    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-model-cache\") pod \"6aa9959a-7c4e-4163-9f57-66721b190d19\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") "
Sep 12 00:41:41.579640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579552    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa9959a-7c4e-4163-9f57-66721b190d19-tls-certs\") pod \"6aa9959a-7c4e-4163-9f57-66721b190d19\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") "
Sep 12 00:41:41.579640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579603    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-home\") pod \"6aa9959a-7c4e-4163-9f57-66721b190d19\" (UID: \"6aa9959a-7c4e-4163-9f57-66721b190d19\") "
Sep 12 00:41:41.579783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579751    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6aa9959a-7c4e-4163-9f57-66721b190d19" (UID: "6aa9959a-7c4e-4163-9f57-66721b190d19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:41.579859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579787    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-model-cache" (OuterVolumeSpecName: "model-cache") pod "6aa9959a-7c4e-4163-9f57-66721b190d19" (UID: "6aa9959a-7c4e-4163-9f57-66721b190d19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:41.579970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579948    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-home" (OuterVolumeSpecName: "home") pod "6aa9959a-7c4e-4163-9f57-66721b190d19" (UID: "6aa9959a-7c4e-4163-9f57-66721b190d19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:41.580098 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.579992    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.580098 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.580037    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.581674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.581650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-dshm" (OuterVolumeSpecName: "dshm") pod "6aa9959a-7c4e-4163-9f57-66721b190d19" (UID: "6aa9959a-7c4e-4163-9f57-66721b190d19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:41:41.581778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.581672    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa9959a-7c4e-4163-9f57-66721b190d19-kube-api-access-8cdcw" (OuterVolumeSpecName: "kube-api-access-8cdcw") pod "6aa9959a-7c4e-4163-9f57-66721b190d19" (UID: "6aa9959a-7c4e-4163-9f57-66721b190d19"). InnerVolumeSpecName "kube-api-access-8cdcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:41:41.581778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.581719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa9959a-7c4e-4163-9f57-66721b190d19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6aa9959a-7c4e-4163-9f57-66721b190d19" (UID: "6aa9959a-7c4e-4163-9f57-66721b190d19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:41:41.681076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.681052    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa9959a-7c4e-4163-9f57-66721b190d19-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.681076 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.681075    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.681229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.681085    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cdcw\" (UniqueName: \"kubernetes.io/projected/6aa9959a-7c4e-4163-9f57-66721b190d19-kube-api-access-8cdcw\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.681229 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.681093    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa9959a-7c4e-4163-9f57-66721b190d19-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.773866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.773793    2562 generic.go:356] "Generic (PLEG): container finished" podID="6aa9959a-7c4e-4163-9f57-66721b190d19" containerID="550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b" exitCode=0
Sep 12 00:41:41.773866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.773839    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"
Sep 12 00:41:41.774010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.773882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9" event={"ID":"6aa9959a-7c4e-4163-9f57-66721b190d19","Type":"ContainerDied","Data":"550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b"}
Sep 12 00:41:41.774010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.773923    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9" event={"ID":"6aa9959a-7c4e-4163-9f57-66721b190d19","Type":"ContainerDied","Data":"8ce6162ddb0f3789b5cf8b1e481571409d0a4ca444d7ab764295446a852d16c7"}
Sep 12 00:41:41.774010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.773942    2562 scope.go:117] "RemoveContainer" containerID="550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b"
Sep 12 00:41:41.783544 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.783529    2562 scope.go:117] "RemoveContainer" containerID="550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b"
Sep 12 00:41:41.783931 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:41:41.783911    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b\": container with ID starting with 550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b not found: ID does not exist" containerID="550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b"
Sep 12 00:41:41.784004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.783939    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b"} err="failed to get container status \"550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b\": rpc error: code = NotFound desc = could not find container \"550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b\": container with ID starting with 550c4cca9672649fdc83849d725303a2f8d55a5d51001d51c7d0d3672ec5565b not found: ID does not exist"
Sep 12 00:41:41.795289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.795269    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"]
Sep 12 00:41:41.798343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.798323    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-139288/e2e-conflict-a-40370e1e-kserve-64f5dc9d68-v55l9"]
Sep 12 00:41:41.957851 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:41.957813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:42.101143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:42.101110    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:41:42.650341 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:42.650309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:42.752181 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:42.752137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:42.853010 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:42.852980    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:43.203587 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:43.203522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:43.299655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:43.299576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:43.571079 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:43.571050    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa9959a-7c4e-4163-9f57-66721b190d19" path="/var/lib/kubelet/pods/6aa9959a-7c4e-4163-9f57-66721b190d19/volumes"
Sep 12 00:41:43.730329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:43.730296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:44.437679 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:44.437636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:45.071763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:45.071730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:45.743133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:45.743095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:46.349418 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:46.349358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:48.308931 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:48.308876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:50.584248 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:50.584208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:53.841157 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:53.841126    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:56.156647 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.156616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:56.175849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.175816    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"]
Sep 12 00:41:56.176126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.176096    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n" podUID="0fd28fc5-bd83-47b9-8606-2723b25b8ba3" containerName="payload-processing" containerID="cri-o://2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10" gracePeriod=30
Sep 12 00:41:56.185813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.185754    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"]
Sep 12 00:41:56.186080 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.186009    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf" podUID="972c3828-67ef-429a-91ea-9722f123acd2" containerName="payload-pre-processing" containerID="cri-o://8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148" gracePeriod=30
Sep 12 00:41:56.255132 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.255100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:56.349787 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.349693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:56.404161 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.404116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:56.428932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.428908    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:41:56.462411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.462381    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:41:56.523032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.523000    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-plugins-config-volume\") pod \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") "
Sep 12 00:41:56.523190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.523094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-kube-api-access-pwvzf\") pod \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\" (UID: \"0fd28fc5-bd83-47b9-8606-2723b25b8ba3\") "
Sep 12 00:41:56.523437 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.523413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fd28fc5-bd83-47b9-8606-2723b25b8ba3" (UID: "0fd28fc5-bd83-47b9-8606-2723b25b8ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:41:56.525030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.525010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-kube-api-access-pwvzf" (OuterVolumeSpecName: "kube-api-access-pwvzf") pod "0fd28fc5-bd83-47b9-8606-2723b25b8ba3" (UID: "0fd28fc5-bd83-47b9-8606-2723b25b8ba3"). InnerVolumeSpecName "kube-api-access-pwvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:41:56.624036 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.624006    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4hzn\" (UniqueName: \"kubernetes.io/projected/972c3828-67ef-429a-91ea-9722f123acd2-kube-api-access-d4hzn\") pod \"972c3828-67ef-429a-91ea-9722f123acd2\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") "
Sep 12 00:41:56.624175 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.624059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972c3828-67ef-429a-91ea-9722f123acd2-plugins-config-volume\") pod \"972c3828-67ef-429a-91ea-9722f123acd2\" (UID: \"972c3828-67ef-429a-91ea-9722f123acd2\") "
Sep 12 00:41:56.624296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.624280    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:56.624346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.624300    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/0fd28fc5-bd83-47b9-8606-2723b25b8ba3-kube-api-access-pwvzf\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:56.624424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.624401    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972c3828-67ef-429a-91ea-9722f123acd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "972c3828-67ef-429a-91ea-9722f123acd2" (UID: "972c3828-67ef-429a-91ea-9722f123acd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:41:56.626198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.626178    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972c3828-67ef-429a-91ea-9722f123acd2-kube-api-access-d4hzn" (OuterVolumeSpecName: "kube-api-access-d4hzn") pod "972c3828-67ef-429a-91ea-9722f123acd2" (UID: "972c3828-67ef-429a-91ea-9722f123acd2"). InnerVolumeSpecName "kube-api-access-d4hzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:41:56.725542 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.725432    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hzn\" (UniqueName: \"kubernetes.io/projected/972c3828-67ef-429a-91ea-9722f123acd2-kube-api-access-d4hzn\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:56.725542 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.725461    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972c3828-67ef-429a-91ea-9722f123acd2-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:41:56.730945 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.730918    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:56.860911 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.860873    2562 generic.go:356] "Generic (PLEG): container finished" podID="0fd28fc5-bd83-47b9-8606-2723b25b8ba3" containerID="2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10" exitCode=0
Sep 12 00:41:56.861095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.860918    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"
Sep 12 00:41:56.861095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.860962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n" event={"ID":"0fd28fc5-bd83-47b9-8606-2723b25b8ba3","Type":"ContainerDied","Data":"2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10"}
Sep 12 00:41:56.861095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.861019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n" event={"ID":"0fd28fc5-bd83-47b9-8606-2723b25b8ba3","Type":"ContainerDied","Data":"bdf153fad3f42bc74934eb3dbf4b3904f7bfa2c497e9b85496ef4220d70bca85"}
Sep 12 00:41:56.861095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.861047    2562 scope.go:117] "RemoveContainer" containerID="2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10"
Sep 12 00:41:56.862529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.862502    2562 generic.go:356] "Generic (PLEG): container finished" podID="972c3828-67ef-429a-91ea-9722f123acd2" containerID="8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148" exitCode=0
Sep 12 00:41:56.862665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.862608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"
Sep 12 00:41:56.862665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.862621    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf" event={"ID":"972c3828-67ef-429a-91ea-9722f123acd2","Type":"ContainerDied","Data":"8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148"}
Sep 12 00:41:56.862665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.862660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf" event={"ID":"972c3828-67ef-429a-91ea-9722f123acd2","Type":"ContainerDied","Data":"a8b39fb032c53763f64c35d3941f04bb6185653d9e89ac5129a3ebe755987874"}
Sep 12 00:41:56.870793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.870776    2562 scope.go:117] "RemoveContainer" containerID="2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10"
Sep 12 00:41:56.871065 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:41:56.871047    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10\": container with ID starting with 2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10 not found: ID does not exist" containerID="2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10"
Sep 12 00:41:56.871143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.871075    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10"} err="failed to get container status \"2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10\": rpc error: code = NotFound desc = could not find container \"2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10\": container with ID starting with 2b6b9f12cecb64e23fe61a5a29153aad5304021981b3bbe622e90ca70b186a10 not found: ID does not exist"
Sep 12 00:41:56.871143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.871098    2562 scope.go:117] "RemoveContainer" containerID="8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148"
Sep 12 00:41:56.879098 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.879080    2562 scope.go:117] "RemoveContainer" containerID="8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148"
Sep 12 00:41:56.879350 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:41:56.879327    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148\": container with ID starting with 8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148 not found: ID does not exist" containerID="8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148"
Sep 12 00:41:56.879428 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.879351    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148"} err="failed to get container status \"8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148\": rpc error: code = NotFound desc = could not find container \"8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148\": container with ID starting with 8c4502f327e395dff4159ac40c4c7c44ce92d616bd6ec7400df6eebc238bd148 not found: ID does not exist"
Sep 12 00:41:56.885696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.885670    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"]
Sep 12 00:41:56.889407 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.889388    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cfdf5d6c-8cd6d5b66-qdmdf"]
Sep 12 00:41:56.899136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.899112    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"]
Sep 12 00:41:56.900966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:56.900946    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cfdf5d6c-7745598cd5-dls2n"]
Sep 12 00:41:57.404248 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:57.404211    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:41:57.570637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:57.570606    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd28fc5-bd83-47b9-8606-2723b25b8ba3" path="/var/lib/kubelet/pods/0fd28fc5-bd83-47b9-8606-2723b25b8ba3/volumes"
Sep 12 00:41:57.570999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:57.570986    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972c3828-67ef-429a-91ea-9722f123acd2" path="/var/lib/kubelet/pods/972c3828-67ef-429a-91ea-9722f123acd2/volumes"
Sep 12 00:41:57.658315 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:57.658285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:58.378776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:58.378741    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:58.736353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:58.736271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:59.274640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:59.274599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:59.695137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:59.695093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:41:59.930258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:41:59.930187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:00.849464 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:00.849429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:01.464984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:01.464953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:02.085364 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:02.085329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:02.853636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:02.853589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:03.591514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.591463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:03.903296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903212    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"]
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903656    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd28fc5-bd83-47b9-8606-2723b25b8ba3" containerName="payload-processing"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903671    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd28fc5-bd83-47b9-8606-2723b25b8ba3" containerName="payload-processing"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903683    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc8359-4a2c-44c1-b387-5840a02b1c48" containerName="main"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903690    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc8359-4a2c-44c1-b387-5840a02b1c48" containerName="main"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903705    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972c3828-67ef-429a-91ea-9722f123acd2" containerName="payload-pre-processing"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903710    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="972c3828-67ef-429a-91ea-9722f123acd2" containerName="payload-pre-processing"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903730    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa9959a-7c4e-4163-9f57-66721b190d19" containerName="main"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903736    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa9959a-7c4e-4163-9f57-66721b190d19" containerName="main"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903794    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="972c3828-67ef-429a-91ea-9722f123acd2" containerName="payload-pre-processing"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903804    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa9959a-7c4e-4163-9f57-66721b190d19" containerName="main"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903811    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc8359-4a2c-44c1-b387-5840a02b1c48" containerName="main"
Sep 12 00:42:03.903844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.903820    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd28fc5-bd83-47b9-8606-2723b25b8ba3" containerName="payload-processing"
Sep 12 00:42:03.908655 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.908635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:03.911350 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.911327    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"kube-root-ca.crt\""
Sep 12 00:42:03.911501 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.911327    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"openshift-service-ca.crt\""
Sep 12 00:42:03.912740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.912719    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"auth-test-model-e88680-kserve-self-signed-certs\""
Sep 12 00:42:03.912844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.912730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"default-dockercfg-bxwk5\""
Sep 12 00:42:03.917751 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.917731    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"]
Sep 12 00:42:03.976072 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:03.976043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:04.094633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.094596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-model-cache\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.094793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.094654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-tmp-dir\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.094793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.094670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40490a82-656f-423d-9f14-8969e5822ef0-tls-certs\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.094793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.094703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-dshm\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.094793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.094721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxqz\" (UniqueName: \"kubernetes.io/projected/40490a82-656f-423d-9f14-8969e5822ef0-kube-api-access-ntxqz\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.094793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.094763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-home\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-model-cache\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-tmp-dir\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40490a82-656f-423d-9f14-8969e5822ef0-tls-certs\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-dshm\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxqz\" (UniqueName: \"kubernetes.io/projected/40490a82-656f-423d-9f14-8969e5822ef0-kube-api-access-ntxqz\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-home\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-model-cache\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.195970 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.195917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-tmp-dir\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.196083 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.196040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-home\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.197888 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.197867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-dshm\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.198037 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.198019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40490a82-656f-423d-9f14-8969e5822ef0-tls-certs\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.202918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.202900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxqz\" (UniqueName: \"kubernetes.io/projected/40490a82-656f-423d-9f14-8969e5822ef0-kube-api-access-ntxqz\") pod \"auth-test-model-e88680-kserve-56f8dd9d99-pjrqr\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") " pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.218921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.218900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:04.494942 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.494214    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"]
Sep 12 00:42:04.501540 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:42:04.501505    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40490a82_656f_423d_9f14_8969e5822ef0.slice/crio-235648386cc42b90112dc40eed7982cd8971757176cfb4929bf708c9335b8cfd WatchSource:0}: Error finding container 235648386cc42b90112dc40eed7982cd8971757176cfb4929bf708c9335b8cfd: Status 404 returned error can't find the container with id 235648386cc42b90112dc40eed7982cd8971757176cfb4929bf708c9335b8cfd
Sep 12 00:42:04.670720 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.670683    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"]
Sep 12 00:42:04.714270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.714236    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:04.908263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.908228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr" event={"ID":"40490a82-656f-423d-9f14-8969e5822ef0","Type":"ContainerStarted","Data":"6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc"}
Sep 12 00:42:04.908678 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.908275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr" event={"ID":"40490a82-656f-423d-9f14-8969e5822ef0","Type":"ContainerStarted","Data":"235648386cc42b90112dc40eed7982cd8971757176cfb4929bf708c9335b8cfd"}
Sep 12 00:42:04.924302 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:04.924257    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr" podStartSLOduration=1.9242406619999999 podStartE2EDuration="1.924240662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:42:04.922918775 +0000 UTC m=+1527.860225294" watchObservedRunningTime="2026-09-12 00:42:04.924240662 +0000 UTC m=+1527.861547183"
Sep 12 00:42:12.809661 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:12.809630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:13.403562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:13.403525    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:42:14.220030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:14.219990    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:14.220030 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:14.220031    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:14.232457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:14.232435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:14.972762 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:14.972734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:42:19.690170 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.690132    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs" podUID="fa5ea539-9326-4f63-98f0-57b38d3c8034" containerName="main" containerID="cri-o://90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4" gracePeriod=45
Sep 12 00:42:19.945820 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.945758    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:42:19.990327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.990288    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa5ea539-9326-4f63-98f0-57b38d3c8034" containerID="90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4" exitCode=0
Sep 12 00:42:19.990327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.990328    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"
Sep 12 00:42:19.990583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.990342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs" event={"ID":"fa5ea539-9326-4f63-98f0-57b38d3c8034","Type":"ContainerDied","Data":"90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4"}
Sep 12 00:42:19.990583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.990391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs" event={"ID":"fa5ea539-9326-4f63-98f0-57b38d3c8034","Type":"ContainerDied","Data":"61e057b4b7c370fc8ceda4ea27026dd93c56c9dede1d652fd237b202d1587497"}
Sep 12 00:42:19.990583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.990421    2562 scope.go:117] "RemoveContainer" containerID="90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4"
Sep 12 00:42:19.999604 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.999590    2562 scope.go:117] "RemoveContainer" containerID="90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4"
Sep 12 00:42:19.999877 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:42:19.999859    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4\": container with ID starting with 90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4 not found: ID does not exist" containerID="90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4"
Sep 12 00:42:19.999932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:19.999884    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4"} err="failed to get container status \"90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4\": rpc error: code = NotFound desc = could not find container \"90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4\": container with ID starting with 90c5cb689ac923983600104e33cafab1c58fdba9a7cda2c12db4f9d0ae969be4 not found: ID does not exist"
Sep 12 00:42:20.044972 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.044948    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5ea539-9326-4f63-98f0-57b38d3c8034-tls-certs\") pod \"fa5ea539-9326-4f63-98f0-57b38d3c8034\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") "
Sep 12 00:42:20.045088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.044978    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-dshm\") pod \"fa5ea539-9326-4f63-98f0-57b38d3c8034\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") "
Sep 12 00:42:20.045088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045035    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-tmp-dir\") pod \"fa5ea539-9326-4f63-98f0-57b38d3c8034\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") "
Sep 12 00:42:20.045088 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045066    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-home\") pod \"fa5ea539-9326-4f63-98f0-57b38d3c8034\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") "
Sep 12 00:42:20.045209 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045145    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-model-cache\") pod \"fa5ea539-9326-4f63-98f0-57b38d3c8034\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") "
Sep 12 00:42:20.045209 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5grtz\" (UniqueName: \"kubernetes.io/projected/fa5ea539-9326-4f63-98f0-57b38d3c8034-kube-api-access-5grtz\") pod \"fa5ea539-9326-4f63-98f0-57b38d3c8034\" (UID: \"fa5ea539-9326-4f63-98f0-57b38d3c8034\") "
Sep 12 00:42:20.045360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa5ea539-9326-4f63-98f0-57b38d3c8034" (UID: "fa5ea539-9326-4f63-98f0-57b38d3c8034"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:42:20.045442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa5ea539-9326-4f63-98f0-57b38d3c8034" (UID: "fa5ea539-9326-4f63-98f0-57b38d3c8034"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:42:20.045442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045398    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-home" (OuterVolumeSpecName: "home") pod "fa5ea539-9326-4f63-98f0-57b38d3c8034" (UID: "fa5ea539-9326-4f63-98f0-57b38d3c8034"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:42:20.045584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045516    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:42:20.045584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045534    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:42:20.045584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.045547    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:42:20.047253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.047224    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-dshm" (OuterVolumeSpecName: "dshm") pod "fa5ea539-9326-4f63-98f0-57b38d3c8034" (UID: "fa5ea539-9326-4f63-98f0-57b38d3c8034"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:42:20.047409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.047390    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5ea539-9326-4f63-98f0-57b38d3c8034-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa5ea539-9326-4f63-98f0-57b38d3c8034" (UID: "fa5ea539-9326-4f63-98f0-57b38d3c8034"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:42:20.047698 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.047667    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5ea539-9326-4f63-98f0-57b38d3c8034-kube-api-access-5grtz" (OuterVolumeSpecName: "kube-api-access-5grtz") pod "fa5ea539-9326-4f63-98f0-57b38d3c8034" (UID: "fa5ea539-9326-4f63-98f0-57b38d3c8034"). InnerVolumeSpecName "kube-api-access-5grtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:42:20.146338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.146307    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grtz\" (UniqueName: \"kubernetes.io/projected/fa5ea539-9326-4f63-98f0-57b38d3c8034-kube-api-access-5grtz\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:42:20.146338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.146335    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5ea539-9326-4f63-98f0-57b38d3c8034-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:42:20.146338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.146345    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5ea539-9326-4f63-98f0-57b38d3c8034-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:42:20.311590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.311559    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"]
Sep 12 00:42:20.315360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:20.315331    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cfdf5d6c/e2e-lifecycle-model-cfdf5d6c-kserve-6979557f75-bxdbs"]
Sep 12 00:42:21.576186 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:21.576153    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5ea539-9326-4f63-98f0-57b38d3c8034" path="/var/lib/kubelet/pods/fa5ea539-9326-4f63-98f0-57b38d3c8034/volumes"
Sep 12 00:42:30.240355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:30.240296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:31.196955 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:31.196920    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:42:31.622271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:31.622237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:40.806096 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:40.806057    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:41.738112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:41.738073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:47.327887 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:47.327846    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:59.303082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.303046    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:42:59.818065 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.818034    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"]
Sep 12 00:42:59.818488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.818454    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5ea539-9326-4f63-98f0-57b38d3c8034" containerName="main"
Sep 12 00:42:59.818488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.818483    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5ea539-9326-4f63-98f0-57b38d3c8034" containerName="main"
Sep 12 00:42:59.818645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.818568    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5ea539-9326-4f63-98f0-57b38d3c8034" containerName="main"
Sep 12 00:42:59.821702 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.821680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:42:59.824231 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.824205    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"default-dockercfg-p2j96\""
Sep 12 00:42:59.824508 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.824464    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"auth-test-model-e37bfb-kserve-self-signed-certs\""
Sep 12 00:42:59.824610 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.824585    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"kube-root-ca.crt\""
Sep 12 00:42:59.824712 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.824693    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"openshift-service-ca.crt\""
Sep 12 00:42:59.831145 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.831121    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"]
Sep 12 00:42:59.846836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.846805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:42:59.908058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.908014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-tmp-dir\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:42:59.908058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.908067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-model-cache\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:42:59.908273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.908089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vp2\" (UniqueName: \"kubernetes.io/projected/2a44ad56-46ca-48cc-a49e-9ab749aec197-kube-api-access-r6vp2\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:42:59.908273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.908138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-home\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:42:59.908273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.908166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44ad56-46ca-48cc-a49e-9ab749aec197-tls-certs\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:42:59.908273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:42:59.908200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-dshm\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.009720 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.009669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-tmp-dir\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.009912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.009735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-model-cache\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.009912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.009765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vp2\" (UniqueName: \"kubernetes.io/projected/2a44ad56-46ca-48cc-a49e-9ab749aec197-kube-api-access-r6vp2\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.009912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.009798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-home\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.009912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.009819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44ad56-46ca-48cc-a49e-9ab749aec197-tls-certs\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.009912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.009859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-dshm\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.010154 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.010130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-tmp-dir\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.010154 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.010144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-model-cache\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.010263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.010206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-home\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.012166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.012145    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-dshm\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.012423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.012406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44ad56-46ca-48cc-a49e-9ab749aec197-tls-certs\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.017408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.017391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vp2\" (UniqueName: \"kubernetes.io/projected/2a44ad56-46ca-48cc-a49e-9ab749aec197-kube-api-access-r6vp2\") pod \"auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.132630 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.132553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:00.404524 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:00.404495    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"]
Sep 12 00:43:00.407174 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:43:00.407145    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a44ad56_46ca_48cc_a49e_9ab749aec197.slice/crio-b7d3000320943ea0d31179f5ab7b074f464c3154c80db9a2f5374fc251db7844 WatchSource:0}: Error finding container b7d3000320943ea0d31179f5ab7b074f464c3154c80db9a2f5374fc251db7844: Status 404 returned error can't find the container with id b7d3000320943ea0d31179f5ab7b074f464c3154c80db9a2f5374fc251db7844
Sep 12 00:43:01.210395 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:01.210363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx" event={"ID":"2a44ad56-46ca-48cc-a49e-9ab749aec197","Type":"ContainerStarted","Data":"a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc"}
Sep 12 00:43:01.210665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:01.210404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx" event={"ID":"2a44ad56-46ca-48cc-a49e-9ab749aec197","Type":"ContainerStarted","Data":"b7d3000320943ea0d31179f5ab7b074f464c3154c80db9a2f5374fc251db7844"}
Sep 12 00:43:01.228320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:01.228270    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx" podStartSLOduration=2.228254196 podStartE2EDuration="2.228254196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:01.225604803 +0000 UTC m=+1584.162911323" watchObservedRunningTime="2026-09-12 00:43:01.228254196 +0000 UTC m=+1584.165560755"
Sep 12 00:43:09.947643 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:09.947586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:10.132864 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:10.132827    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:10.133027 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:10.132878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:10.145265 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:10.145237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:10.268925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:10.268841    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:43:11.280645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:11.280614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:11.602305 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:11.602264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:11.702610 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:11.702573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:12.354829 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:12.354799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:12.452841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:12.452805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:12.553168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:12.553141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:13.893912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:13.893876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:19.543575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:19.543539    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:19.643745 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:19.643713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:20.240572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:20.240540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:22.707401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:22.707363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:43:23.559530 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:23.559490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:23.659925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:23.659886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:26.562979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:26.562945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:26.665773 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:26.665738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:26.763410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:26.763377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:31.923995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:31.923916    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:37.279197 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:37.279168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:37.500752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:37.500713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:37.736712 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:37.736680    2562 scope.go:117] "RemoveContainer" containerID="7e489bfe79f84d579658c90e02d1d3982d3244a47d89edc4b6f5927fc595435c"
Sep 12 00:43:44.231892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:44.231856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:46.005273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:46.005228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:43:58.468814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:43:58.468779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:04.375318 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:04.375274    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:04.435000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:04.434965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:04.746197 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:04.746114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:05.058625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:05.058534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:05.448117 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:05.448079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:07.455832 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:07.455775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:08.113414 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:08.113373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:08.208006 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:08.207972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:08.946694 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:08.946655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:09.313026 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:09.312947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:44:09.722746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:09.722707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:10.492810 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:10.492770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:11.269813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:11.269779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:12.144036 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.143995    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"]
Sep 12 00:44:12.147591 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.147567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.150209 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.150163    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-829959-serving-cert\""
Sep 12 00:44:12.154516 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.154465    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"]
Sep 12 00:44:12.168391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.168362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:12.198380 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.198338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bfp\" (UniqueName: \"kubernetes.io/projected/833eab5d-c934-467d-9442-d5d1c1e6356e-kube-api-access-q4bfp\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.198608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.198442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.198608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.198503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833eab5d-c934-467d-9442-d5d1c1e6356e-openshift-service-ca\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.198608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.198538    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-metrics-certs\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.269395 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.269353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:12.299242 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.299194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bfp\" (UniqueName: \"kubernetes.io/projected/833eab5d-c934-467d-9442-d5d1c1e6356e-kube-api-access-q4bfp\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.299431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.299293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.299431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.299315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833eab5d-c934-467d-9442-d5d1c1e6356e-openshift-service-ca\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.299431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.299337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-metrics-certs\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.299643 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:44:12.299445    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-829959-serving-cert: secret "maas-api-e2e-api-a-829959-serving-cert" not found
Sep 12 00:44:12.299643 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:44:12.299540    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls podName:833eab5d-c934-467d-9442-d5d1c1e6356e nodeName:}" failed. No retries permitted until 2026-09-12 00:44:12.799520122 +0000 UTC m=+1655.736826630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls") pod "maas-api-e2e-api-a-829959-77c5d75bff-thqzk" (UID: "833eab5d-c934-467d-9442-d5d1c1e6356e") : secret "maas-api-e2e-api-a-829959-serving-cert" not found
Sep 12 00:44:12.300014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.299984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833eab5d-c934-467d-9442-d5d1c1e6356e-openshift-service-ca\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.302019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.301996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-metrics-certs\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.307446 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.307422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bfp\" (UniqueName: \"kubernetes.io/projected/833eab5d-c934-467d-9442-d5d1c1e6356e-kube-api-access-q4bfp\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.369669 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.369628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:12.626938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.626904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:12.725074 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.725039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:12.732512 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.732466    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"]
Sep 12 00:44:12.736043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.736025    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:12.738459 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.738435    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-829959-dockercfg-d2fsr\""
Sep 12 00:44:12.738686 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.738669    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-829959\""
Sep 12 00:44:12.747198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.747154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"]
Sep 12 00:44:12.802893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.802857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.803097 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.802922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f00bf0d-7445-42b1-bffc-fbce063bd41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:12.803097 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.802949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhh2\" (UniqueName: \"kubernetes.io/projected/9f00bf0d-7445-42b1-bffc-fbce063bd41a-kube-api-access-whhh2\") pod \"payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:12.805285 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.805265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls\") pod \"maas-api-e2e-api-a-829959-77c5d75bff-thqzk\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:12.903657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.903551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f00bf0d-7445-42b1-bffc-fbce063bd41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:12.903657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.903593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhh2\" (UniqueName: \"kubernetes.io/projected/9f00bf0d-7445-42b1-bffc-fbce063bd41a-kube-api-access-whhh2\") pod \"payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:12.904306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.904281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f00bf0d-7445-42b1-bffc-fbce063bd41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:12.911505 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:12.911465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhh2\" (UniqueName: \"kubernetes.io/projected/9f00bf0d-7445-42b1-bffc-fbce063bd41a-kube-api-access-whhh2\") pod \"payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:13.047978 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.047938    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:13.060183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.060135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:13.318386 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.318352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"]
Sep 12 00:44:13.320498 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:44:13.320448    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f00bf0d_7445_42b1_bffc_fbce063bd41a.slice/crio-13f98ae9225d289aa2299ceb97d3f340a481b091e6e399db4e328c57c1dfb068 WatchSource:0}: Error finding container 13f98ae9225d289aa2299ceb97d3f340a481b091e6e399db4e328c57c1dfb068: Status 404 returned error can't find the container with id 13f98ae9225d289aa2299ceb97d3f340a481b091e6e399db4e328c57c1dfb068
Sep 12 00:44:13.338960 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.338936    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"]
Sep 12 00:44:13.589813 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.589729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk" event={"ID":"833eab5d-c934-467d-9442-d5d1c1e6356e","Type":"ContainerStarted","Data":"9b188baf5c525c66d96991f5bcb9ed630e846414fec053cbebf995b731695ce2"}
Sep 12 00:44:13.591263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.591235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk" event={"ID":"9f00bf0d-7445-42b1-bffc-fbce063bd41a","Type":"ContainerStarted","Data":"26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0"}
Sep 12 00:44:13.591401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.591283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk" event={"ID":"9f00bf0d-7445-42b1-bffc-fbce063bd41a","Type":"ContainerStarted","Data":"13f98ae9225d289aa2299ceb97d3f340a481b091e6e399db4e328c57c1dfb068"}
Sep 12 00:44:13.591401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.591362    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:13.606448 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:13.606405    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk" podStartSLOduration=1.6063898170000002 podStartE2EDuration="1.606389817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:13.604987733 +0000 UTC m=+1656.542294254" watchObservedRunningTime="2026-09-12 00:44:13.606389817 +0000 UTC m=+1656.543696337"
Sep 12 00:44:14.598732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:14.598691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk" event={"ID":"833eab5d-c934-467d-9442-d5d1c1e6356e","Type":"ContainerStarted","Data":"1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66"}
Sep 12 00:44:14.599156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:14.598913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:14.617187 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:14.617135    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk" podStartSLOduration=2.204962338 podStartE2EDuration="2.617121637s" totalImagesPullingTime="412.159299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:14.614951672 +0000 UTC m=+1657.552258193" watchObservedRunningTime="2026-09-12 00:44:14.617121637 +0000 UTC m=+1657.554428157"
Sep 12 00:44:20.609527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:20.609460    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:44:24.601085 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:24.601049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:44:30.172701 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:30.172658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:30.787320 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:30.787261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:31.538344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:31.538310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:32.556271 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:32.556213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:33.098636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:33.098596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:33.262854 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:33.262809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:33.560949 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:33.560873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:33.869932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:33.869893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:34.178078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:34.178004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:34.303337 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:34.303301    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:44:34.560017 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:34.559938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:34.947350 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:34.947245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:35.169649 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:35.169581    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"]
Sep 12 00:44:35.248340 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:35.248254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:35.666202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:35.666171    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:36.680343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:36.680311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:37.296517 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:37.296467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:37.392806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:37.392771    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:37.691382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:37.691339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:38.456635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:38.456599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:39.364129 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.364095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:39.464971 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.464938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:39.564695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.564658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:39.849916 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.849885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:39.946760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.946726    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"]
Sep 12 00:44:39.950822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.950795    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:39.951357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.951337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:39.953198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.953175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-593845\""
Sep 12 00:44:39.961234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:39.961209    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"]
Sep 12 00:44:40.082511 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.082452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5971a5-90aa-40e4-9231-54e70b554be7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.082670 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.082608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kw7\" (UniqueName: \"kubernetes.io/projected/4d5971a5-90aa-40e4-9231-54e70b554be7-kube-api-access-v6kw7\") pod \"payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.183969 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.183877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kw7\" (UniqueName: \"kubernetes.io/projected/4d5971a5-90aa-40e4-9231-54e70b554be7-kube-api-access-v6kw7\") pod \"payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.183969 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.183962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5971a5-90aa-40e4-9231-54e70b554be7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.184649 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.184629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5971a5-90aa-40e4-9231-54e70b554be7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.192054 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.192026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kw7\" (UniqueName: \"kubernetes.io/projected/4d5971a5-90aa-40e4-9231-54e70b554be7-kube-api-access-v6kw7\") pod \"payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.263270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.263238    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.290651 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.290607    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:40.538731 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.538699    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"]
Sep 12 00:44:40.540454 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:44:40.540428    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5971a5_90aa_40e4_9231_54e70b554be7.slice/crio-733095f76dc6ea04e4cad59614b52fedddc01b2cba1fc7c92fc2a166d1e0d60b WatchSource:0}: Error finding container 733095f76dc6ea04e4cad59614b52fedddc01b2cba1fc7c92fc2a166d1e0d60b: Status 404 returned error can't find the container with id 733095f76dc6ea04e4cad59614b52fedddc01b2cba1fc7c92fc2a166d1e0d60b
Sep 12 00:44:40.737653 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.737607    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448" event={"ID":"4d5971a5-90aa-40e4-9231-54e70b554be7","Type":"ContainerStarted","Data":"8a9156a1f48122ae70261b6457830182834686e35aae84f92ebf24d0a8e989fc"}
Sep 12 00:44:40.737852 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.737681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448" event={"ID":"4d5971a5-90aa-40e4-9231-54e70b554be7","Type":"ContainerStarted","Data":"733095f76dc6ea04e4cad59614b52fedddc01b2cba1fc7c92fc2a166d1e0d60b"}
Sep 12 00:44:40.737852 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.737709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:40.754705 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:40.754649    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448" podStartSLOduration=1.7546334799999999 podStartE2EDuration="1.75463348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:40.752722428 +0000 UTC m=+1683.690028950" watchObservedRunningTime="2026-09-12 00:44:40.75463348 +0000 UTC m=+1683.691939999"
Sep 12 00:44:41.044247 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:41.044210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:41.786085 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:41.786050    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:42.528658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:42.528625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:43.354305 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:43.354275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:48.944685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:48.944647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:48.962695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:48.962662    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"]
Sep 12 00:44:48.962927 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:48.962899    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h" podUID="fdc4378e-2441-4fe1-a9ac-77a1d595e507" containerName="payload-processing" containerID="cri-o://f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172" gracePeriod=30
Sep 12 00:44:48.972194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:48.972165    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"]
Sep 12 00:44:48.972509 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:48.972448    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85" podUID="42d2d637-9abe-4449-9efe-fbf4305af8d0" containerName="payload-pre-processing" containerID="cri-o://0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25" gracePeriod=30
Sep 12 00:44:49.004546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.004520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.068766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.068730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.166041 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.166010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.225818 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.225794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.249722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.249683    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:44:49.257386 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.257367    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:44:49.271183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.271154    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwm7t\" (UniqueName: \"kubernetes.io/projected/42d2d637-9abe-4449-9efe-fbf4305af8d0-kube-api-access-hwm7t\") pod \"42d2d637-9abe-4449-9efe-fbf4305af8d0\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") "
Sep 12 00:44:49.271311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.271207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc72q\" (UniqueName: \"kubernetes.io/projected/fdc4378e-2441-4fe1-a9ac-77a1d595e507-kube-api-access-lc72q\") pod \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") "
Sep 12 00:44:49.271311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.271239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2d637-9abe-4449-9efe-fbf4305af8d0-plugins-config-volume\") pod \"42d2d637-9abe-4449-9efe-fbf4305af8d0\" (UID: \"42d2d637-9abe-4449-9efe-fbf4305af8d0\") "
Sep 12 00:44:49.271426 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.271317    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc4378e-2441-4fe1-a9ac-77a1d595e507-plugins-config-volume\") pod \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\" (UID: \"fdc4378e-2441-4fe1-a9ac-77a1d595e507\") "
Sep 12 00:44:49.272021 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.271964    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d2d637-9abe-4449-9efe-fbf4305af8d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42d2d637-9abe-4449-9efe-fbf4305af8d0" (UID: "42d2d637-9abe-4449-9efe-fbf4305af8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:44:49.272021 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.271969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc4378e-2441-4fe1-a9ac-77a1d595e507-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdc4378e-2441-4fe1-a9ac-77a1d595e507" (UID: "fdc4378e-2441-4fe1-a9ac-77a1d595e507"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:44:49.274105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.274083    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d2d637-9abe-4449-9efe-fbf4305af8d0-kube-api-access-hwm7t" (OuterVolumeSpecName: "kube-api-access-hwm7t") pod "42d2d637-9abe-4449-9efe-fbf4305af8d0" (UID: "42d2d637-9abe-4449-9efe-fbf4305af8d0"). InnerVolumeSpecName "kube-api-access-hwm7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:44:49.274242 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.274211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc4378e-2441-4fe1-a9ac-77a1d595e507-kube-api-access-lc72q" (OuterVolumeSpecName: "kube-api-access-lc72q") pod "fdc4378e-2441-4fe1-a9ac-77a1d595e507" (UID: "fdc4378e-2441-4fe1-a9ac-77a1d595e507"). InnerVolumeSpecName "kube-api-access-lc72q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:44:49.373207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.373173    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc72q\" (UniqueName: \"kubernetes.io/projected/fdc4378e-2441-4fe1-a9ac-77a1d595e507-kube-api-access-lc72q\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:49.373207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.373200    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2d637-9abe-4449-9efe-fbf4305af8d0-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:49.373207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.373212    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc4378e-2441-4fe1-a9ac-77a1d595e507-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:49.373445 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.373221    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwm7t\" (UniqueName: \"kubernetes.io/projected/42d2d637-9abe-4449-9efe-fbf4305af8d0-kube-api-access-hwm7t\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:49.547405 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.547318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.753908 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.753875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:49.796682 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.796645    2562 generic.go:356] "Generic (PLEG): container finished" podID="42d2d637-9abe-4449-9efe-fbf4305af8d0" containerID="0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25" exitCode=0
Sep 12 00:44:49.796841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.796700    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"
Sep 12 00:44:49.796841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.796725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85" event={"ID":"42d2d637-9abe-4449-9efe-fbf4305af8d0","Type":"ContainerDied","Data":"0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25"}
Sep 12 00:44:49.796841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.796778    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85" event={"ID":"42d2d637-9abe-4449-9efe-fbf4305af8d0","Type":"ContainerDied","Data":"ddee3e3bd406d2d66e77e431f560cd87ac5897b14e1366e9511c77df07a1f463"}
Sep 12 00:44:49.796841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.796823    2562 scope.go:117] "RemoveContainer" containerID="0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25"
Sep 12 00:44:49.798304 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.798286    2562 generic.go:356] "Generic (PLEG): container finished" podID="fdc4378e-2441-4fe1-a9ac-77a1d595e507" containerID="f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172" exitCode=0
Sep 12 00:44:49.798412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.798327    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"
Sep 12 00:44:49.798490 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.798321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h" event={"ID":"fdc4378e-2441-4fe1-a9ac-77a1d595e507","Type":"ContainerDied","Data":"f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172"}
Sep 12 00:44:49.798551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.798503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h" event={"ID":"fdc4378e-2441-4fe1-a9ac-77a1d595e507","Type":"ContainerDied","Data":"72a1709c96c307aa810aaa735c13bbfac917ce90f8fe6fd531a471f1ec0260af"}
Sep 12 00:44:49.808731 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.808666    2562 scope.go:117] "RemoveContainer" containerID="0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25"
Sep 12 00:44:49.809155 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:44:49.809133    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25\": container with ID starting with 0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25 not found: ID does not exist" containerID="0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25"
Sep 12 00:44:49.809224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.809165    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25"} err="failed to get container status \"0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25\": rpc error: code = NotFound desc = could not find container \"0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25\": container with ID starting with 0d8791ff22b9521203791faf2b000f174bdcb08aef1c34c426a430ff3d075c25 not found: ID does not exist"
Sep 12 00:44:49.809224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.809183    2562 scope.go:117] "RemoveContainer" containerID="f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172"
Sep 12 00:44:49.817613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.817594    2562 scope.go:117] "RemoveContainer" containerID="f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172"
Sep 12 00:44:49.817880 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:44:49.817864    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172\": container with ID starting with f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172 not found: ID does not exist" containerID="f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172"
Sep 12 00:44:49.817939 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.817885    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172"} err="failed to get container status \"f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172\": rpc error: code = NotFound desc = could not find container \"f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172\": container with ID starting with f989d3cd6a2b7232bec9a7d3c073a2bf7d85841b7aa8c3d5f70363bf664e8172 not found: ID does not exist"
Sep 12 00:44:49.818668 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.818648    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"]
Sep 12 00:44:49.822577 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.822558    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-139288-664fd6dffc-9bb8h"]
Sep 12 00:44:49.832559 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.832537    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"]
Sep 12 00:44:49.835989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:49.835970    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-139288-666ffbb86b-6jl85"]
Sep 12 00:44:50.188789 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.188745    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx" podUID="4fcf4a89-1132-4203-ab1f-873ae5733c25" containerName="main" containerID="cri-o://3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873" gracePeriod=45
Sep 12 00:44:50.438354 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.438327    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:44:50.471774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.471702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:50.483214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-model-cache\") pod \"4fcf4a89-1132-4203-ab1f-873ae5733c25\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") "
Sep 12 00:44:50.483378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483222    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-tmp-dir\") pod \"4fcf4a89-1132-4203-ab1f-873ae5733c25\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") "
Sep 12 00:44:50.483378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483247    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rhz5\" (UniqueName: \"kubernetes.io/projected/4fcf4a89-1132-4203-ab1f-873ae5733c25-kube-api-access-8rhz5\") pod \"4fcf4a89-1132-4203-ab1f-873ae5733c25\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") "
Sep 12 00:44:50.483378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483342    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-home\") pod \"4fcf4a89-1132-4203-ab1f-873ae5733c25\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") "
Sep 12 00:44:50.483546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483411    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-dshm\") pod \"4fcf4a89-1132-4203-ab1f-873ae5733c25\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") "
Sep 12 00:44:50.483546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf4a89-1132-4203-ab1f-873ae5733c25-tls-certs\") pod \"4fcf4a89-1132-4203-ab1f-873ae5733c25\" (UID: \"4fcf4a89-1132-4203-ab1f-873ae5733c25\") "
Sep 12 00:44:50.483546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483487    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fcf4a89-1132-4203-ab1f-873ae5733c25" (UID: "4fcf4a89-1132-4203-ab1f-873ae5733c25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:44:50.483546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483516    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fcf4a89-1132-4203-ab1f-873ae5733c25" (UID: "4fcf4a89-1132-4203-ab1f-873ae5733c25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:44:50.483765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483677    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-home" (OuterVolumeSpecName: "home") pod "4fcf4a89-1132-4203-ab1f-873ae5733c25" (UID: "4fcf4a89-1132-4203-ab1f-873ae5733c25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:44:50.483841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483825    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:50.483912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483846    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:50.483912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.483884    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:50.485666 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.485641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcf4a89-1132-4203-ab1f-873ae5733c25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fcf4a89-1132-4203-ab1f-873ae5733c25" (UID: "4fcf4a89-1132-4203-ab1f-873ae5733c25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:44:50.485766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.485641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-dshm" (OuterVolumeSpecName: "dshm") pod "4fcf4a89-1132-4203-ab1f-873ae5733c25" (UID: "4fcf4a89-1132-4203-ab1f-873ae5733c25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:44:50.485766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.485718    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcf4a89-1132-4203-ab1f-873ae5733c25-kube-api-access-8rhz5" (OuterVolumeSpecName: "kube-api-access-8rhz5") pod "4fcf4a89-1132-4203-ab1f-873ae5733c25" (UID: "4fcf4a89-1132-4203-ab1f-873ae5733c25"). InnerVolumeSpecName "kube-api-access-8rhz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:44:50.584606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.584567    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf4a89-1132-4203-ab1f-873ae5733c25-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:50.584606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.584597    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf4a89-1132-4203-ab1f-873ae5733c25-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:50.584606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.584608    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rhz5\" (UniqueName: \"kubernetes.io/projected/4fcf4a89-1132-4203-ab1f-873ae5733c25-kube-api-access-8rhz5\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:44:50.805450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.805353    2562 generic.go:356] "Generic (PLEG): container finished" podID="4fcf4a89-1132-4203-ab1f-873ae5733c25" containerID="3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873" exitCode=0
Sep 12 00:44:50.805450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.805400    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"
Sep 12 00:44:50.805725 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.805443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx" event={"ID":"4fcf4a89-1132-4203-ab1f-873ae5733c25","Type":"ContainerDied","Data":"3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873"}
Sep 12 00:44:50.805725 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.805529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx" event={"ID":"4fcf4a89-1132-4203-ab1f-873ae5733c25","Type":"ContainerDied","Data":"fbf771b70cb1e559bf8643c5da32ece410124f106b28c6aea72e4666cdf27fbd"}
Sep 12 00:44:50.805725 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.805555    2562 scope.go:117] "RemoveContainer" containerID="3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873"
Sep 12 00:44:50.814878 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.814860    2562 scope.go:117] "RemoveContainer" containerID="3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873"
Sep 12 00:44:50.815144 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:44:50.815126    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873\": container with ID starting with 3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873 not found: ID does not exist" containerID="3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873"
Sep 12 00:44:50.815192 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.815152    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873"} err="failed to get container status \"3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873\": rpc error: code = NotFound desc = could not find container \"3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873\": container with ID starting with 3f0c1c85e35c730fc6cae3894a36f5c8a609d579fa756f63f278001dcd073873 not found: ID does not exist"
Sep 12 00:44:50.826450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.826419    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"]
Sep 12 00:44:50.832049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:50.832029    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-139288/simulator-w4-139288-kserve-88d578f88-5bwtx"]
Sep 12 00:44:51.365376 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:51.365327    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:51.571734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:51.571705    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d2d637-9abe-4449-9efe-fbf4305af8d0" path="/var/lib/kubelet/pods/42d2d637-9abe-4449-9efe-fbf4305af8d0/volumes"
Sep 12 00:44:51.572118 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:51.572105    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fcf4a89-1132-4203-ab1f-873ae5733c25" path="/var/lib/kubelet/pods/4fcf4a89-1132-4203-ab1f-873ae5733c25/volumes"
Sep 12 00:44:51.572428 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:51.572417    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc4378e-2441-4fe1-a9ac-77a1d595e507" path="/var/lib/kubelet/pods/fdc4378e-2441-4fe1-a9ac-77a1d595e507/volumes"
Sep 12 00:44:51.746156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:51.746064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:44:52.119016 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:52.118979    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:52.879069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:52.879036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:53.655255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:53.655218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:54.630142 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:54.630104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.386959 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:55.386924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.678555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:55.678460    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.777801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:55.777762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:55.960739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:55.960653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:56.238768 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:56.238683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.057033 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:57.056999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.177576 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:57.177545    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:44:57.735608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:44:57.735577    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:00.144028 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.143941    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"]
Sep 12 00:45:00.144447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144342    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcf4a89-1132-4203-ab1f-873ae5733c25" containerName="main"
Sep 12 00:45:00.144447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144354    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcf4a89-1132-4203-ab1f-873ae5733c25" containerName="main"
Sep 12 00:45:00.144447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144369    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc4378e-2441-4fe1-a9ac-77a1d595e507" containerName="payload-processing"
Sep 12 00:45:00.144447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144375    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc4378e-2441-4fe1-a9ac-77a1d595e507" containerName="payload-processing"
Sep 12 00:45:00.144447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144383    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d2d637-9abe-4449-9efe-fbf4305af8d0" containerName="payload-pre-processing"
Sep 12 00:45:00.144447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144388    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d2d637-9abe-4449-9efe-fbf4305af8d0" containerName="payload-pre-processing"
Sep 12 00:45:00.144706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144455    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d2d637-9abe-4449-9efe-fbf4305af8d0" containerName="payload-pre-processing"
Sep 12 00:45:00.144706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144463    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcf4a89-1132-4203-ab1f-873ae5733c25" containerName="main"
Sep 12 00:45:00.144706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.144487    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc4378e-2441-4fe1-a9ac-77a1d595e507" containerName="payload-processing"
Sep 12 00:45:00.149202 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.149171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"
Sep 12 00:45:00.152420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.152385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:45:00.154706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.154680    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"]
Sep 12 00:45:00.160588 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.160558    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"]
Sep 12 00:45:00.163177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.163154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"
Sep 12 00:45:00.164664 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.164647    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"
Sep 12 00:45:00.176502 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.175455    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"]
Sep 12 00:45:00.198166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.198134    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"
Sep 12 00:45:00.446274 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.446247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"]
Sep 12 00:45:00.448048 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:45:00.448020    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65195835_4f66_4e39_9400_86eeaa894220.slice/crio-c396da91f7cd8665ff2727d22441a59e60f38a0bb492b936cc3a78a2fb60353e WatchSource:0}: Error finding container c396da91f7cd8665ff2727d22441a59e60f38a0bb492b936cc3a78a2fb60353e: Status 404 returned error can't find the container with id c396da91f7cd8665ff2727d22441a59e60f38a0bb492b936cc3a78a2fb60353e
Sep 12 00:45:00.480767 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.480745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"]
Sep 12 00:45:00.482664 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:45:00.482637    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428715cc_5025_4296_a5b4_2899fc27b7b9.slice/crio-245763b35e7caf0319c43d9b6eb55f1469e9bd0abbdf7be691827764f8c350b0 WatchSource:0}: Error finding container 245763b35e7caf0319c43d9b6eb55f1469e9bd0abbdf7be691827764f8c350b0: Status 404 returned error can't find the container with id 245763b35e7caf0319c43d9b6eb55f1469e9bd0abbdf7be691827764f8c350b0
Sep 12 00:45:00.652387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.652329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:00.867004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.866966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b" event={"ID":"428715cc-5025-4296-a5b4-2899fc27b7b9","Type":"ContainerStarted","Data":"5bbc4022449c69a09ddabf5a07ea7084c38a1749f49eb8efb30f77ebe4993552"}
Sep 12 00:45:00.867172 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.867060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b" event={"ID":"428715cc-5025-4296-a5b4-2899fc27b7b9","Type":"ContainerStarted","Data":"245763b35e7caf0319c43d9b6eb55f1469e9bd0abbdf7be691827764f8c350b0"}
Sep 12 00:45:00.868585 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.868559    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc" event={"ID":"65195835-4f66-4e39-9400-86eeaa894220","Type":"ContainerStarted","Data":"43863a2b15d472871a2dd2b23336b61ce19d615004e964e1693c50c0d9f532f1"}
Sep 12 00:45:00.868697 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.868604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc" event={"ID":"65195835-4f66-4e39-9400-86eeaa894220","Type":"ContainerStarted","Data":"c396da91f7cd8665ff2727d22441a59e60f38a0bb492b936cc3a78a2fb60353e"}
Sep 12 00:45:00.881395 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.881345    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b" podStartSLOduration=0.881330617 podStartE2EDuration="881.330617ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:00.880271372 +0000 UTC m=+1703.817577890" watchObservedRunningTime="2026-09-12 00:45:00.881330617 +0000 UTC m=+1703.818637137"
Sep 12 00:45:00.895957 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:00.895900    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc" podStartSLOduration=0.895883018 podStartE2EDuration="895.883018ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:00.894303487 +0000 UTC m=+1703.831610008" watchObservedRunningTime="2026-09-12 00:45:00.895883018 +0000 UTC m=+1703.833189583"
Sep 12 00:45:01.688213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:01.688131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:01.875270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:01.875236    2562 generic.go:356] "Generic (PLEG): container finished" podID="428715cc-5025-4296-a5b4-2899fc27b7b9" containerID="5bbc4022449c69a09ddabf5a07ea7084c38a1749f49eb8efb30f77ebe4993552" exitCode=0
Sep 12 00:45:01.875421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:01.875324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b" event={"ID":"428715cc-5025-4296-a5b4-2899fc27b7b9","Type":"ContainerDied","Data":"5bbc4022449c69a09ddabf5a07ea7084c38a1749f49eb8efb30f77ebe4993552"}
Sep 12 00:45:01.876860 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:01.876839    2562 generic.go:356] "Generic (PLEG): container finished" podID="65195835-4f66-4e39-9400-86eeaa894220" containerID="43863a2b15d472871a2dd2b23336b61ce19d615004e964e1693c50c0d9f532f1" exitCode=0
Sep 12 00:45:01.876978 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:01.876868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc" event={"ID":"65195835-4f66-4e39-9400-86eeaa894220","Type":"ContainerDied","Data":"43863a2b15d472871a2dd2b23336b61ce19d615004e964e1693c50c0d9f532f1"}
Sep 12 00:45:02.451932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:02.451895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:03.017380 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.017354    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"
Sep 12 00:45:03.052022 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.051992    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"
Sep 12 00:45:03.173342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.173265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:03.888672 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.888588    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"
Sep 12 00:45:03.888672 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.888596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b" event={"ID":"428715cc-5025-4296-a5b4-2899fc27b7b9","Type":"ContainerDied","Data":"245763b35e7caf0319c43d9b6eb55f1469e9bd0abbdf7be691827764f8c350b0"}
Sep 12 00:45:03.888672 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.888670    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245763b35e7caf0319c43d9b6eb55f1469e9bd0abbdf7be691827764f8c350b0"
Sep 12 00:45:03.890152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.890132    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"
Sep 12 00:45:03.890286 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.890190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc" event={"ID":"65195835-4f66-4e39-9400-86eeaa894220","Type":"ContainerDied","Data":"c396da91f7cd8665ff2727d22441a59e60f38a0bb492b936cc3a78a2fb60353e"}
Sep 12 00:45:03.890286 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.890234    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c396da91f7cd8665ff2727d22441a59e60f38a0bb492b936cc3a78a2fb60353e"
Sep 12 00:45:03.901650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.901622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:45:03.997893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:03.997858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:04.828950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:04.828909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:05.539156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:05.539117    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:06.183017 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:06.182975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:07.564979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:07.564944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:10.550009 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:10.549970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:11.311880 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:11.311849    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:11.361265 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:11.361234    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:12.813042 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:12.813009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.208402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.208366    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"]
Sep 12 00:45:16.208872 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.208698    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk" podUID="833eab5d-c934-467d-9442-d5d1c1e6356e" containerName="maas-api" containerID="cri-o://1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66" gracePeriod=30
Sep 12 00:45:16.235381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.235355    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"]
Sep 12 00:45:16.236900 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.236871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.238541 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.238514    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-829959-29819565-2wm5b"]
Sep 12 00:45:16.265585 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.265552    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"]
Sep 12 00:45:16.265822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.265795    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk" podUID="9f00bf0d-7445-42b1-bffc-fbce063bd41a" containerName="payload-pre-processing" containerID="cri-o://26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0" gracePeriod=30
Sep 12 00:45:16.301425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.301398    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.402802 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.402734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.503963 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.503934    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.538083 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.538053    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:45:16.541296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.541258    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:45:16.647074 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647046    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-metrics-certs\") pod \"833eab5d-c934-467d-9442-d5d1c1e6356e\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") "
Sep 12 00:45:16.647277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647088    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whhh2\" (UniqueName: \"kubernetes.io/projected/9f00bf0d-7445-42b1-bffc-fbce063bd41a-kube-api-access-whhh2\") pod \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") "
Sep 12 00:45:16.647277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls\") pod \"833eab5d-c934-467d-9442-d5d1c1e6356e\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") "
Sep 12 00:45:16.647277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647188    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4bfp\" (UniqueName: \"kubernetes.io/projected/833eab5d-c934-467d-9442-d5d1c1e6356e-kube-api-access-q4bfp\") pod \"833eab5d-c934-467d-9442-d5d1c1e6356e\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") "
Sep 12 00:45:16.647277 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647263    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833eab5d-c934-467d-9442-d5d1c1e6356e-openshift-service-ca\") pod \"833eab5d-c934-467d-9442-d5d1c1e6356e\" (UID: \"833eab5d-c934-467d-9442-d5d1c1e6356e\") "
Sep 12 00:45:16.647518 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f00bf0d-7445-42b1-bffc-fbce063bd41a-plugins-config-volume\") pod \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\" (UID: \"9f00bf0d-7445-42b1-bffc-fbce063bd41a\") "
Sep 12 00:45:16.647887 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.647833    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833eab5d-c934-467d-9442-d5d1c1e6356e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "833eab5d-c934-467d-9442-d5d1c1e6356e" (UID: "833eab5d-c934-467d-9442-d5d1c1e6356e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:16.648064 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.648036    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f00bf0d-7445-42b1-bffc-fbce063bd41a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f00bf0d-7445-42b1-bffc-fbce063bd41a" (UID: "9f00bf0d-7445-42b1-bffc-fbce063bd41a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:16.649619 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.649581    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "833eab5d-c934-467d-9442-d5d1c1e6356e" (UID: "833eab5d-c934-467d-9442-d5d1c1e6356e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:16.649805 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.649771    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f00bf0d-7445-42b1-bffc-fbce063bd41a-kube-api-access-whhh2" (OuterVolumeSpecName: "kube-api-access-whhh2") pod "9f00bf0d-7445-42b1-bffc-fbce063bd41a" (UID: "9f00bf0d-7445-42b1-bffc-fbce063bd41a"). InnerVolumeSpecName "kube-api-access-whhh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:16.649935 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.649825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "833eab5d-c934-467d-9442-d5d1c1e6356e" (UID: "833eab5d-c934-467d-9442-d5d1c1e6356e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:16.649935 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.649929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833eab5d-c934-467d-9442-d5d1c1e6356e-kube-api-access-q4bfp" (OuterVolumeSpecName: "kube-api-access-q4bfp") pod "833eab5d-c934-467d-9442-d5d1c1e6356e" (UID: "833eab5d-c934-467d-9442-d5d1c1e6356e"). InnerVolumeSpecName "kube-api-access-q4bfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:16.748827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.748556    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:16.748827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.748591    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhh2\" (UniqueName: \"kubernetes.io/projected/9f00bf0d-7445-42b1-bffc-fbce063bd41a-kube-api-access-whhh2\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:16.748827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.748687    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833eab5d-c934-467d-9442-d5d1c1e6356e-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:16.748827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.748712    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4bfp\" (UniqueName: \"kubernetes.io/projected/833eab5d-c934-467d-9442-d5d1c1e6356e-kube-api-access-q4bfp\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:16.748827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.748728    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833eab5d-c934-467d-9442-d5d1c1e6356e-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:16.748827 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.748744    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f00bf0d-7445-42b1-bffc-fbce063bd41a-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:16.779980 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.779948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.880394 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.880361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:16.967681 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.967645    2562 generic.go:356] "Generic (PLEG): container finished" podID="833eab5d-c934-467d-9442-d5d1c1e6356e" containerID="1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66" exitCode=0
Sep 12 00:45:16.967849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.967699    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"
Sep 12 00:45:16.967849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.967731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk" event={"ID":"833eab5d-c934-467d-9442-d5d1c1e6356e","Type":"ContainerDied","Data":"1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66"}
Sep 12 00:45:16.967849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.967786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk" event={"ID":"833eab5d-c934-467d-9442-d5d1c1e6356e","Type":"ContainerDied","Data":"9b188baf5c525c66d96991f5bcb9ed630e846414fec053cbebf995b731695ce2"}
Sep 12 00:45:16.967849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.967814    2562 scope.go:117] "RemoveContainer" containerID="1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66"
Sep 12 00:45:16.969194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.969168    2562 generic.go:356] "Generic (PLEG): container finished" podID="9f00bf0d-7445-42b1-bffc-fbce063bd41a" containerID="26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0" exitCode=0
Sep 12 00:45:16.969303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.969217    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"
Sep 12 00:45:16.969303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.969251    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk" event={"ID":"9f00bf0d-7445-42b1-bffc-fbce063bd41a","Type":"ContainerDied","Data":"26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0"}
Sep 12 00:45:16.969412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.969312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk" event={"ID":"9f00bf0d-7445-42b1-bffc-fbce063bd41a","Type":"ContainerDied","Data":"13f98ae9225d289aa2299ceb97d3f340a481b091e6e399db4e328c57c1dfb068"}
Sep 12 00:45:16.978290 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.978270    2562 scope.go:117] "RemoveContainer" containerID="1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66"
Sep 12 00:45:16.978601 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:45:16.978581    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66\": container with ID starting with 1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66 not found: ID does not exist" containerID="1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66"
Sep 12 00:45:16.978706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.978622    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66"} err="failed to get container status \"1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66\": rpc error: code = NotFound desc = could not find container \"1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66\": container with ID starting with 1df573535a695190c2d33c48bb618da86f143efbd1763d30828b757136941c66 not found: ID does not exist"
Sep 12 00:45:16.978706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.978645    2562 scope.go:117] "RemoveContainer" containerID="26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0"
Sep 12 00:45:16.987134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.987116    2562 scope.go:117] "RemoveContainer" containerID="26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0"
Sep 12 00:45:16.987393 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:45:16.987371    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0\": container with ID starting with 26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0 not found: ID does not exist" containerID="26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0"
Sep 12 00:45:16.987455 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.987404    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0"} err="failed to get container status \"26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0\": rpc error: code = NotFound desc = could not find container \"26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0\": container with ID starting with 26ac75397602ee5dbe9cefedcc107e3361ef923ae82e361e36c7e6826e8003b0 not found: ID does not exist"
Sep 12 00:45:16.993171 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.993142    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"]
Sep 12 00:45:16.997337 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:16.997314    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-829959-77c5d75bff-thqzk"]
Sep 12 00:45:17.007771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:17.007719    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"]
Sep 12 00:45:17.014144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:17.014117    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-829959-89589dff4-vjfjk"]
Sep 12 00:45:17.570099 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:17.570054    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428715cc-5025-4296-a5b4-2899fc27b7b9" path="/var/lib/kubelet/pods/428715cc-5025-4296-a5b4-2899fc27b7b9/volumes"
Sep 12 00:45:17.570640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:17.570619    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833eab5d-c934-467d-9442-d5d1c1e6356e" path="/var/lib/kubelet/pods/833eab5d-c934-467d-9442-d5d1c1e6356e/volumes"
Sep 12 00:45:17.571012 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:17.570993    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f00bf0d-7445-42b1-bffc-fbce063bd41a" path="/var/lib/kubelet/pods/9f00bf0d-7445-42b1-bffc-fbce063bd41a/volumes"
Sep 12 00:45:17.620381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:17.620346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:18.436447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:18.436414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.357069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:19.357037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:19.980200 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:19.980165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:20.733165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:20.733130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:22.650646 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:22.650613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:22.711346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:22.711307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:22.920899 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:22.920819    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:23.949128 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:23.949083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:24.645812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:24.645772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:25.336632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:25.336602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:25.974751 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:25.974715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:26.830070 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:26.830032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:27.370686 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:27.370650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:27.412097 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:27.412067    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:45:36.053856 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:36.053824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:38.893691 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:38.893659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:38.898044 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:38.898017    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"]
Sep 12 00:45:38.899845 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:38.899824    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-593845-29819565-nfklc"]
Sep 12 00:45:38.930105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:38.930078    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"]
Sep 12 00:45:38.930375 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:38.930349    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448" podUID="4d5971a5-90aa-40e4-9231-54e70b554be7" containerName="payload-pre-processing" containerID="cri-o://8a9156a1f48122ae70261b6457830182834686e35aae84f92ebf24d0a8e989fc" gracePeriod=30
Sep 12 00:45:38.990888 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:38.990854    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:39.075879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.075843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:39.090462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.090433    2562 generic.go:356] "Generic (PLEG): container finished" podID="4d5971a5-90aa-40e4-9231-54e70b554be7" containerID="8a9156a1f48122ae70261b6457830182834686e35aae84f92ebf24d0a8e989fc" exitCode=0
Sep 12 00:45:39.090624 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.090501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448" event={"ID":"4d5971a5-90aa-40e4-9231-54e70b554be7","Type":"ContainerDied","Data":"8a9156a1f48122ae70261b6457830182834686e35aae84f92ebf24d0a8e989fc"}
Sep 12 00:45:39.131867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.131831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:39.188567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.188544    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:45:39.263776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.263745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5971a5-90aa-40e4-9231-54e70b554be7-plugins-config-volume\") pod \"4d5971a5-90aa-40e4-9231-54e70b554be7\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") "
Sep 12 00:45:39.263940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.263799    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6kw7\" (UniqueName: \"kubernetes.io/projected/4d5971a5-90aa-40e4-9231-54e70b554be7-kube-api-access-v6kw7\") pod \"4d5971a5-90aa-40e4-9231-54e70b554be7\" (UID: \"4d5971a5-90aa-40e4-9231-54e70b554be7\") "
Sep 12 00:45:39.264146 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.264124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5971a5-90aa-40e4-9231-54e70b554be7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d5971a5-90aa-40e4-9231-54e70b554be7" (UID: "4d5971a5-90aa-40e4-9231-54e70b554be7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:39.265840 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.265821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5971a5-90aa-40e4-9231-54e70b554be7-kube-api-access-v6kw7" (OuterVolumeSpecName: "kube-api-access-v6kw7") pod "4d5971a5-90aa-40e4-9231-54e70b554be7" (UID: "4d5971a5-90aa-40e4-9231-54e70b554be7"). InnerVolumeSpecName "kube-api-access-v6kw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:39.364905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.364872    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5971a5-90aa-40e4-9231-54e70b554be7-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:39.364905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.364899    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kw7\" (UniqueName: \"kubernetes.io/projected/4d5971a5-90aa-40e4-9231-54e70b554be7-kube-api-access-v6kw7\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:39.429324 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.429248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:39.529621 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.529592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:39.571395 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:39.571366    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65195835-4f66-4e39-9400-86eeaa894220" path="/var/lib/kubelet/pods/65195835-4f66-4e39-9400-86eeaa894220/volumes"
Sep 12 00:45:40.096706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.096677    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"
Sep 12 00:45:40.096706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.096690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448" event={"ID":"4d5971a5-90aa-40e4-9231-54e70b554be7","Type":"ContainerDied","Data":"733095f76dc6ea04e4cad59614b52fedddc01b2cba1fc7c92fc2a166d1e0d60b"}
Sep 12 00:45:40.097158 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.096752    2562 scope.go:117] "RemoveContainer" containerID="8a9156a1f48122ae70261b6457830182834686e35aae84f92ebf24d0a8e989fc"
Sep 12 00:45:40.112681 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.112655    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"]
Sep 12 00:45:40.115982 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.115960    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-593845-6976cbf4f8-b5448"]
Sep 12 00:45:40.242306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.242276    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:40.917104 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:40.917068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:41.549895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:41.549857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:41.570061 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:41.570026    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5971a5-90aa-40e4-9231-54e70b554be7" path="/var/lib/kubelet/pods/4d5971a5-90aa-40e4-9231-54e70b554be7/volumes"
Sep 12 00:45:42.338048 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:42.338014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:43.301732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:43.301696    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:45:45.180027 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:45.179989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:45.238645 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:45.238608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:45.455958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:45.455882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:45.840383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:45.840348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:45.970739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:45.970706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:46.847087 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:46.847056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:47.655207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:47.655170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:48.311966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:48.311929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:49.042238 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:49.042200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:49.450764 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:49.450730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"]
Sep 12 00:45:49.468527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:49.468481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:49.835807 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:49.835755    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:50.317369 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:50.317339    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"]
Sep 12 00:45:50.342677 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:50.342650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:53.199707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:53.199676    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:45:56.282049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.282010    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"]
Sep 12 00:45:56.282506 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.282260    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58" podUID="ad029918-7abd-4dee-b03e-a94bbbbae7c7" containerName="maas-api" containerID="cri-o://3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60" gracePeriod=30
Sep 12 00:45:56.299222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.299107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:56.339154 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.339122    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"]
Sep 12 00:45:56.339430 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.339395    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6" podUID="f78890d1-1c85-484e-a5e1-9b43c894c2fe" containerName="payload-pre-processing" containerID="cri-o://149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d" gracePeriod=30
Sep 12 00:45:56.396668 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.396636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:56.487632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.487596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:56.546025 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.545949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:56.591765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.591737    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:45:56.618644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.618621    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:45:56.730801 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.730764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-maas-api-tls\") pod \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") "
Sep 12 00:45:56.730985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.730836    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/ad029918-7abd-4dee-b03e-a94bbbbae7c7-kube-api-access-49lrj\") pod \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") "
Sep 12 00:45:56.730985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.730872    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78890d1-1c85-484e-a5e1-9b43c894c2fe-plugins-config-volume\") pod \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") "
Sep 12 00:45:56.730985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.730910    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-metrics-certs\") pod \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") "
Sep 12 00:45:56.730985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.730931    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad029918-7abd-4dee-b03e-a94bbbbae7c7-openshift-service-ca\") pod \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\" (UID: \"ad029918-7abd-4dee-b03e-a94bbbbae7c7\") "
Sep 12 00:45:56.730985 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.730971    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55ssv\" (UniqueName: \"kubernetes.io/projected/f78890d1-1c85-484e-a5e1-9b43c894c2fe-kube-api-access-55ssv\") pod \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\" (UID: \"f78890d1-1c85-484e-a5e1-9b43c894c2fe\") "
Sep 12 00:45:56.731337 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.731304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78890d1-1c85-484e-a5e1-9b43c894c2fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f78890d1-1c85-484e-a5e1-9b43c894c2fe" (UID: "f78890d1-1c85-484e-a5e1-9b43c894c2fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:56.731415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.731315    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad029918-7abd-4dee-b03e-a94bbbbae7c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad029918-7abd-4dee-b03e-a94bbbbae7c7" (UID: "ad029918-7abd-4dee-b03e-a94bbbbae7c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:56.733362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.733326    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad029918-7abd-4dee-b03e-a94bbbbae7c7" (UID: "ad029918-7abd-4dee-b03e-a94bbbbae7c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:56.733362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.733333    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78890d1-1c85-484e-a5e1-9b43c894c2fe-kube-api-access-55ssv" (OuterVolumeSpecName: "kube-api-access-55ssv") pod "f78890d1-1c85-484e-a5e1-9b43c894c2fe" (UID: "f78890d1-1c85-484e-a5e1-9b43c894c2fe"). InnerVolumeSpecName "kube-api-access-55ssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:56.733524 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.733438    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad029918-7abd-4dee-b03e-a94bbbbae7c7-kube-api-access-49lrj" (OuterVolumeSpecName: "kube-api-access-49lrj") pod "ad029918-7abd-4dee-b03e-a94bbbbae7c7" (UID: "ad029918-7abd-4dee-b03e-a94bbbbae7c7"). InnerVolumeSpecName "kube-api-access-49lrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:56.733524 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.733449    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad029918-7abd-4dee-b03e-a94bbbbae7c7" (UID: "ad029918-7abd-4dee-b03e-a94bbbbae7c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:56.832635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.832608    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:56.832635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.832631    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad029918-7abd-4dee-b03e-a94bbbbae7c7-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:56.832786 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.832642    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ssv\" (UniqueName: \"kubernetes.io/projected/f78890d1-1c85-484e-a5e1-9b43c894c2fe-kube-api-access-55ssv\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:56.832786 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.832651    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad029918-7abd-4dee-b03e-a94bbbbae7c7-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:56.832786 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.832659    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49lrj\" (UniqueName: \"kubernetes.io/projected/ad029918-7abd-4dee-b03e-a94bbbbae7c7-kube-api-access-49lrj\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:56.832786 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.832667    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78890d1-1c85-484e-a5e1-9b43c894c2fe-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:45:56.864226 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.864200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:56.970583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:56.970554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:57.200313 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.200223    2562 generic.go:356] "Generic (PLEG): container finished" podID="ad029918-7abd-4dee-b03e-a94bbbbae7c7" containerID="3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60" exitCode=0
Sep 12 00:45:57.200313 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.200264    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"
Sep 12 00:45:57.200551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.200313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58" event={"ID":"ad029918-7abd-4dee-b03e-a94bbbbae7c7","Type":"ContainerDied","Data":"3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60"}
Sep 12 00:45:57.200551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.200362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58" event={"ID":"ad029918-7abd-4dee-b03e-a94bbbbae7c7","Type":"ContainerDied","Data":"40e8f0284abaa749c6bf9ae74ec439b090f74f4a31d010b37ee72c7940393bfc"}
Sep 12 00:45:57.200551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.200387    2562 scope.go:117] "RemoveContainer" containerID="3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60"
Sep 12 00:45:57.201729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.201708    2562 generic.go:356] "Generic (PLEG): container finished" podID="f78890d1-1c85-484e-a5e1-9b43c894c2fe" containerID="149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d" exitCode=0
Sep 12 00:45:57.201841 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.201761    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"
Sep 12 00:45:57.201895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.201757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6" event={"ID":"f78890d1-1c85-484e-a5e1-9b43c894c2fe","Type":"ContainerDied","Data":"149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d"}
Sep 12 00:45:57.201895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.201888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6" event={"ID":"f78890d1-1c85-484e-a5e1-9b43c894c2fe","Type":"ContainerDied","Data":"8f7247005981c9275369df6fb82c35b529ec69294219c4af1296c2c23d6ae09e"}
Sep 12 00:45:57.210911 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.210897    2562 scope.go:117] "RemoveContainer" containerID="3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60"
Sep 12 00:45:57.211240 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:45:57.211212    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60\": container with ID starting with 3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60 not found: ID does not exist" containerID="3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60"
Sep 12 00:45:57.211309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.211254    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60"} err="failed to get container status \"3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60\": rpc error: code = NotFound desc = could not find container \"3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60\": container with ID starting with 3da7882532e22fc551e99a72c4e393221f136c95bfdf21e94e26cbbf53452f60 not found: ID does not exist"
Sep 12 00:45:57.211309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.211281    2562 scope.go:117] "RemoveContainer" containerID="149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d"
Sep 12 00:45:57.219392 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.219376    2562 scope.go:117] "RemoveContainer" containerID="149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d"
Sep 12 00:45:57.219644 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:45:57.219629    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d\": container with ID starting with 149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d not found: ID does not exist" containerID="149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d"
Sep 12 00:45:57.219698 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.219648    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d"} err="failed to get container status \"149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d\": rpc error: code = NotFound desc = could not find container \"149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d\": container with ID starting with 149bde7d4e9106a08a244d4f1f4f4887d97eefa3d5812b782a157a096c4d6f2d not found: ID does not exist"
Sep 12 00:45:57.222900 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.222877    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"]
Sep 12 00:45:57.226211 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.226191    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0d3a66-54dc6bbd84-cqp58"]
Sep 12 00:45:57.235334 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.235314    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"]
Sep 12 00:45:57.239208 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.239187    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0d3a66-9874b58d6-m2bp6"]
Sep 12 00:45:57.571844 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.571811    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad029918-7abd-4dee-b03e-a94bbbbae7c7" path="/var/lib/kubelet/pods/ad029918-7abd-4dee-b03e-a94bbbbae7c7/volumes"
Sep 12 00:45:57.572191 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.572168    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78890d1-1c85-484e-a5e1-9b43c894c2fe" path="/var/lib/kubelet/pods/f78890d1-1c85-484e-a5e1-9b43c894c2fe/volumes"
Sep 12 00:45:57.598884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:57.598860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:58.367123 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:58.367091    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:45:59.141842 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:45:59.141803    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:02.539921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:02.539884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:02.602603 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:02.602569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:02.789739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:02.789709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:03.134694 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.134657    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"]
Sep 12 00:46:03.135020 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135007    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad029918-7abd-4dee-b03e-a94bbbbae7c7" containerName="maas-api"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135021    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad029918-7abd-4dee-b03e-a94bbbbae7c7" containerName="maas-api"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135030    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f00bf0d-7445-42b1-bffc-fbce063bd41a" containerName="payload-pre-processing"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135037    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f00bf0d-7445-42b1-bffc-fbce063bd41a" containerName="payload-pre-processing"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135048    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5971a5-90aa-40e4-9231-54e70b554be7" containerName="payload-pre-processing"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135054    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5971a5-90aa-40e4-9231-54e70b554be7" containerName="payload-pre-processing"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135061    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78890d1-1c85-484e-a5e1-9b43c894c2fe" containerName="payload-pre-processing"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135066    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78890d1-1c85-484e-a5e1-9b43c894c2fe" containerName="payload-pre-processing"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135073    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428715cc-5025-4296-a5b4-2899fc27b7b9" containerName="cleanup"
Sep 12 00:46:03.135082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135078    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="428715cc-5025-4296-a5b4-2899fc27b7b9" containerName="cleanup"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135090    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65195835-4f66-4e39-9400-86eeaa894220" containerName="cleanup"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135095    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="65195835-4f66-4e39-9400-86eeaa894220" containerName="cleanup"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135104    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833eab5d-c934-467d-9442-d5d1c1e6356e" containerName="maas-api"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135109    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="833eab5d-c934-467d-9442-d5d1c1e6356e" containerName="maas-api"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135165    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="65195835-4f66-4e39-9400-86eeaa894220" containerName="cleanup"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135173    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="428715cc-5025-4296-a5b4-2899fc27b7b9" containerName="cleanup"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135178    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f00bf0d-7445-42b1-bffc-fbce063bd41a" containerName="payload-pre-processing"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135186    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5971a5-90aa-40e4-9231-54e70b554be7" containerName="payload-pre-processing"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135192    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad029918-7abd-4dee-b03e-a94bbbbae7c7" containerName="maas-api"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135198    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78890d1-1c85-484e-a5e1-9b43c894c2fe" containerName="payload-pre-processing"
Sep 12 00:46:03.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.135204    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="833eab5d-c934-467d-9442-d5d1c1e6356e" containerName="maas-api"
Sep 12 00:46:03.139922 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.139901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:03.153589 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.143665    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:46:03.153589 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.147033    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"]
Sep 12 00:46:03.192616 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.192591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e469902-fc5d-4f45-9b1b-4497452d3eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n\" (UID: \"7e469902-fc5d-4f45-9b1b-4497452d3eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:03.293879 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.293845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e469902-fc5d-4f45-9b1b-4497452d3eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n\" (UID: \"7e469902-fc5d-4f45-9b1b-4497452d3eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:03.294547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.294527    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e469902-fc5d-4f45-9b1b-4497452d3eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n\" (UID: \"7e469902-fc5d-4f45-9b1b-4497452d3eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:03.361818 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.361779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:03.455447 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.455370    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:03.728120 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:03.728095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"]
Sep 12 00:46:03.729939 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:46:03.729912    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e469902_fc5d_4f45_9b1b_4497452d3eca.slice/crio-0e4b3c89e77ba05c7948be076d1d387ae1b7f2fb51128f91ead1267f4d392737 WatchSource:0}: Error finding container 0e4b3c89e77ba05c7948be076d1d387ae1b7f2fb51128f91ead1267f4d392737: Status 404 returned error can't find the container with id 0e4b3c89e77ba05c7948be076d1d387ae1b7f2fb51128f91ead1267f4d392737
Sep 12 00:46:04.064208 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.064161    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:04.242714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.242627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n" event={"ID":"7e469902-fc5d-4f45-9b1b-4497452d3eca","Type":"ContainerStarted","Data":"cb2ed5304ae9f1de6443aad1746f10c9b7cb6fa618c03dcd3bedfdf86b8dd4d3"}
Sep 12 00:46:04.242714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.242681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n" event={"ID":"7e469902-fc5d-4f45-9b1b-4497452d3eca","Type":"ContainerStarted","Data":"0e4b3c89e77ba05c7948be076d1d387ae1b7f2fb51128f91ead1267f4d392737"}
Sep 12 00:46:04.258987 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.258926    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n" podStartSLOduration=1.25890778 podStartE2EDuration="1.25890778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:04.258198485 +0000 UTC m=+1767.195505004" watchObservedRunningTime="2026-09-12 00:46:04.25890778 +0000 UTC m=+1767.196214300"
Sep 12 00:46:04.470735 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.470682    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr" podUID="40490a82-656f-423d-9f14-8969e5822ef0" containerName="main" containerID="cri-o://6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc" gracePeriod=45
Sep 12 00:46:04.717629 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.717604    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:46:04.805584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805500    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntxqz\" (UniqueName: \"kubernetes.io/projected/40490a82-656f-423d-9f14-8969e5822ef0-kube-api-access-ntxqz\") pod \"40490a82-656f-423d-9f14-8969e5822ef0\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") "
Sep 12 00:46:04.805584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805562    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-model-cache\") pod \"40490a82-656f-423d-9f14-8969e5822ef0\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") "
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805594    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-tmp-dir\") pod \"40490a82-656f-423d-9f14-8969e5822ef0\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") "
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805623    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-home\") pod \"40490a82-656f-423d-9f14-8969e5822ef0\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") "
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-dshm\") pod \"40490a82-656f-423d-9f14-8969e5822ef0\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") "
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40490a82-656f-423d-9f14-8969e5822ef0-tls-certs\") pod \"40490a82-656f-423d-9f14-8969e5822ef0\" (UID: \"40490a82-656f-423d-9f14-8969e5822ef0\") "
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805855    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40490a82-656f-423d-9f14-8969e5822ef0" (UID: "40490a82-656f-423d-9f14-8969e5822ef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805869    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-model-cache" (OuterVolumeSpecName: "model-cache") pod "40490a82-656f-423d-9f14-8969e5822ef0" (UID: "40490a82-656f-423d-9f14-8969e5822ef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.805929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-home" (OuterVolumeSpecName: "home") pod "40490a82-656f-423d-9f14-8969e5822ef0" (UID: "40490a82-656f-423d-9f14-8969e5822ef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:04.806043 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.806044    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:04.806313 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.806058    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:04.806313 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.806067    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:04.807734 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.807702    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40490a82-656f-423d-9f14-8969e5822ef0-kube-api-access-ntxqz" (OuterVolumeSpecName: "kube-api-access-ntxqz") pod "40490a82-656f-423d-9f14-8969e5822ef0" (UID: "40490a82-656f-423d-9f14-8969e5822ef0"). InnerVolumeSpecName "kube-api-access-ntxqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:04.807823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.807749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40490a82-656f-423d-9f14-8969e5822ef0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40490a82-656f-423d-9f14-8969e5822ef0" (UID: "40490a82-656f-423d-9f14-8969e5822ef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:04.807823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.807749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-dshm" (OuterVolumeSpecName: "dshm") pod "40490a82-656f-423d-9f14-8969e5822ef0" (UID: "40490a82-656f-423d-9f14-8969e5822ef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:04.861254 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.861221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:04.906769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.906736    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40490a82-656f-423d-9f14-8969e5822ef0-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:04.906769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.906763    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntxqz\" (UniqueName: \"kubernetes.io/projected/40490a82-656f-423d-9f14-8969e5822ef0-kube-api-access-ntxqz\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:04.906938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:04.906776    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40490a82-656f-423d-9f14-8969e5822ef0-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:05.249551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.249508    2562 generic.go:356] "Generic (PLEG): container finished" podID="40490a82-656f-423d-9f14-8969e5822ef0" containerID="6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc" exitCode=0
Sep 12 00:46:05.249761 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.249605    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"
Sep 12 00:46:05.249761 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.249622    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr" event={"ID":"40490a82-656f-423d-9f14-8969e5822ef0","Type":"ContainerDied","Data":"6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc"}
Sep 12 00:46:05.249761 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.249677    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr" event={"ID":"40490a82-656f-423d-9f14-8969e5822ef0","Type":"ContainerDied","Data":"235648386cc42b90112dc40eed7982cd8971757176cfb4929bf708c9335b8cfd"}
Sep 12 00:46:05.249761 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.249702    2562 scope.go:117] "RemoveContainer" containerID="6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc"
Sep 12 00:46:05.251613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.251586    2562 generic.go:356] "Generic (PLEG): container finished" podID="7e469902-fc5d-4f45-9b1b-4497452d3eca" containerID="cb2ed5304ae9f1de6443aad1746f10c9b7cb6fa618c03dcd3bedfdf86b8dd4d3" exitCode=0
Sep 12 00:46:05.251796 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.251627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n" event={"ID":"7e469902-fc5d-4f45-9b1b-4497452d3eca","Type":"ContainerDied","Data":"cb2ed5304ae9f1de6443aad1746f10c9b7cb6fa618c03dcd3bedfdf86b8dd4d3"}
Sep 12 00:46:05.259790 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.259774    2562 scope.go:117] "RemoveContainer" containerID="6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc"
Sep 12 00:46:05.260085 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:46:05.260064    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc\": container with ID starting with 6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc not found: ID does not exist" containerID="6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc"
Sep 12 00:46:05.260134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.260093    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc"} err="failed to get container status \"6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc\": rpc error: code = NotFound desc = could not find container \"6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc\": container with ID starting with 6f2eb12467f96aeb4360d623ec5ef027c17c0c25133f2607e1203c94e1117bdc not found: ID does not exist"
Sep 12 00:46:05.281951 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.281926    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"]
Sep 12 00:46:05.284620 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.284598    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/auth-test-model-e88680-kserve-56f8dd9d99-pjrqr"]
Sep 12 00:46:05.338162 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.338129    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx" podUID="2a44ad56-46ca-48cc-a49e-9ab749aec197" containerName="main" containerID="cri-o://a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc" gracePeriod=45
Sep 12 00:46:05.508777 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.508742    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:46:05.559415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.559380    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:05.571837 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.571809    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40490a82-656f-423d-9f14-8969e5822ef0" path="/var/lib/kubelet/pods/40490a82-656f-423d-9f14-8969e5822ef0/volumes"
Sep 12 00:46:05.595429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.595408    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:46:05.612000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.611977    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-tmp-dir\") pod \"2a44ad56-46ca-48cc-a49e-9ab749aec197\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") "
Sep 12 00:46:05.612115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612009    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-model-cache\") pod \"2a44ad56-46ca-48cc-a49e-9ab749aec197\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") "
Sep 12 00:46:05.612115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-dshm\") pod \"2a44ad56-46ca-48cc-a49e-9ab749aec197\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") "
Sep 12 00:46:05.612115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612081    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-home\") pod \"2a44ad56-46ca-48cc-a49e-9ab749aec197\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") "
Sep 12 00:46:05.612295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612143    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6vp2\" (UniqueName: \"kubernetes.io/projected/2a44ad56-46ca-48cc-a49e-9ab749aec197-kube-api-access-r6vp2\") pod \"2a44ad56-46ca-48cc-a49e-9ab749aec197\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") "
Sep 12 00:46:05.612295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612204    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44ad56-46ca-48cc-a49e-9ab749aec197-tls-certs\") pod \"2a44ad56-46ca-48cc-a49e-9ab749aec197\" (UID: \"2a44ad56-46ca-48cc-a49e-9ab749aec197\") "
Sep 12 00:46:05.612295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a44ad56-46ca-48cc-a49e-9ab749aec197" (UID: "2a44ad56-46ca-48cc-a49e-9ab749aec197"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:05.612456 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a44ad56-46ca-48cc-a49e-9ab749aec197" (UID: "2a44ad56-46ca-48cc-a49e-9ab749aec197"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:05.612544 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-home" (OuterVolumeSpecName: "home") pod "2a44ad56-46ca-48cc-a49e-9ab749aec197" (UID: "2a44ad56-46ca-48cc-a49e-9ab749aec197"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:05.612651 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612622    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:05.612724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612650    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:05.612724 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.612664    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:05.614910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.614881    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a44ad56-46ca-48cc-a49e-9ab749aec197-kube-api-access-r6vp2" (OuterVolumeSpecName: "kube-api-access-r6vp2") pod "2a44ad56-46ca-48cc-a49e-9ab749aec197" (UID: "2a44ad56-46ca-48cc-a49e-9ab749aec197"). InnerVolumeSpecName "kube-api-access-r6vp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:05.615523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.615394    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-dshm" (OuterVolumeSpecName: "dshm") pod "2a44ad56-46ca-48cc-a49e-9ab749aec197" (UID: "2a44ad56-46ca-48cc-a49e-9ab749aec197"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:05.615523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.615407    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a44ad56-46ca-48cc-a49e-9ab749aec197-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a44ad56-46ca-48cc-a49e-9ab749aec197" (UID: "2a44ad56-46ca-48cc-a49e-9ab749aec197"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:05.713819 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.713789    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a44ad56-46ca-48cc-a49e-9ab749aec197-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:05.713819 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.713814    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a44ad56-46ca-48cc-a49e-9ab749aec197-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:05.713819 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:05.713824    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vp2\" (UniqueName: \"kubernetes.io/projected/2a44ad56-46ca-48cc-a49e-9ab749aec197-kube-api-access-r6vp2\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:06.260199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.260167    2562 generic.go:356] "Generic (PLEG): container finished" podID="2a44ad56-46ca-48cc-a49e-9ab749aec197" containerID="a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc" exitCode=0
Sep 12 00:46:06.260746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.260215    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"
Sep 12 00:46:06.260746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.260260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx" event={"ID":"2a44ad56-46ca-48cc-a49e-9ab749aec197","Type":"ContainerDied","Data":"a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc"}
Sep 12 00:46:06.260746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.260305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx" event={"ID":"2a44ad56-46ca-48cc-a49e-9ab749aec197","Type":"ContainerDied","Data":"b7d3000320943ea0d31179f5ab7b074f464c3154c80db9a2f5374fc251db7844"}
Sep 12 00:46:06.260746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.260322    2562 scope.go:117] "RemoveContainer" containerID="a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc"
Sep 12 00:46:06.270777 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.270760    2562 scope.go:117] "RemoveContainer" containerID="a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc"
Sep 12 00:46:06.271075 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:46:06.271049    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc\": container with ID starting with a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc not found: ID does not exist" containerID="a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc"
Sep 12 00:46:06.271140 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.271078    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc"} err="failed to get container status \"a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc\": rpc error: code = NotFound desc = could not find container \"a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc\": container with ID starting with a092ddadfc932900ab91201ad6cd30adab1328588dca5f4fb0f5da8a265730bc not found: ID does not exist"
Sep 12 00:46:06.283143 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.283116    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"]
Sep 12 00:46:06.286912 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.286891    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/auth-test-model-e37bfb-kserve-8666c8c6c5-t8wqx"]
Sep 12 00:46:06.369780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.369741    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:06.390562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.390530    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:06.420688 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.420657    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e469902-fc5d-4f45-9b1b-4497452d3eca-maas-api-service-ca\") pod \"7e469902-fc5d-4f45-9b1b-4497452d3eca\" (UID: \"7e469902-fc5d-4f45-9b1b-4497452d3eca\") "
Sep 12 00:46:06.421019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.420992    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e469902-fc5d-4f45-9b1b-4497452d3eca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e469902-fc5d-4f45-9b1b-4497452d3eca" (UID: "7e469902-fc5d-4f45-9b1b-4497452d3eca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:06.521344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:06.521273    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e469902-fc5d-4f45-9b1b-4497452d3eca-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:07.268312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:07.268279    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"
Sep 12 00:46:07.268312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:07.268292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n" event={"ID":"7e469902-fc5d-4f45-9b1b-4497452d3eca","Type":"ContainerDied","Data":"0e4b3c89e77ba05c7948be076d1d387ae1b7f2fb51128f91ead1267f4d392737"}
Sep 12 00:46:07.268749 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:07.268333    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e4b3c89e77ba05c7948be076d1d387ae1b7f2fb51128f91ead1267f4d392737"
Sep 12 00:46:07.571183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:07.571150    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a44ad56-46ca-48cc-a49e-9ab749aec197" path="/var/lib/kubelet/pods/2a44ad56-46ca-48cc-a49e-9ab749aec197/volumes"
Sep 12 00:46:13.557966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.557930    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"]
Sep 12 00:46:13.558527 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.558195    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" podUID="a8d189e5-143c-4698-8118-11fcd3daa98e" containerName="maas-api" containerID="cri-o://fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3" gracePeriod=30
Sep 12 00:46:13.587944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.587917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:13.688630 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.688604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:13.765547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.765515    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"]
Sep 12 00:46:13.767337 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.767312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:13.767491 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.767439    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f12ab7-kp27n"]
Sep 12 00:46:13.801455 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.801436    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:46:13.826346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.826324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:13.885889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.885863    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-267r4\" (UniqueName: \"kubernetes.io/projected/a8d189e5-143c-4698-8118-11fcd3daa98e-kube-api-access-267r4\") pod \"a8d189e5-143c-4698-8118-11fcd3daa98e\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") "
Sep 12 00:46:13.886008 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.885959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-metrics-certs\") pod \"a8d189e5-143c-4698-8118-11fcd3daa98e\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") "
Sep 12 00:46:13.886008 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.885980    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d189e5-143c-4698-8118-11fcd3daa98e-openshift-service-ca\") pod \"a8d189e5-143c-4698-8118-11fcd3daa98e\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") "
Sep 12 00:46:13.886176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.886023    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls\") pod \"a8d189e5-143c-4698-8118-11fcd3daa98e\" (UID: \"a8d189e5-143c-4698-8118-11fcd3daa98e\") "
Sep 12 00:46:13.886404 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.886368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d189e5-143c-4698-8118-11fcd3daa98e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8d189e5-143c-4698-8118-11fcd3daa98e" (UID: "a8d189e5-143c-4698-8118-11fcd3daa98e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:13.888066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.888038    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a8d189e5-143c-4698-8118-11fcd3daa98e" (UID: "a8d189e5-143c-4698-8118-11fcd3daa98e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:13.888066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.888052    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d189e5-143c-4698-8118-11fcd3daa98e-kube-api-access-267r4" (OuterVolumeSpecName: "kube-api-access-267r4") pod "a8d189e5-143c-4698-8118-11fcd3daa98e" (UID: "a8d189e5-143c-4698-8118-11fcd3daa98e"). InnerVolumeSpecName "kube-api-access-267r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:13.888186 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.888109    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8d189e5-143c-4698-8118-11fcd3daa98e" (UID: "a8d189e5-143c-4698-8118-11fcd3daa98e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:46:13.987600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.987565    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d189e5-143c-4698-8118-11fcd3daa98e-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:13.987600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.987597    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:13.987769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.987609    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-267r4\" (UniqueName: \"kubernetes.io/projected/a8d189e5-143c-4698-8118-11fcd3daa98e-kube-api-access-267r4\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:13.987769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:13.987618    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8d189e5-143c-4698-8118-11fcd3daa98e-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:46:14.153409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.153316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:14.254634 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.254596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:14.303257 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.303220    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8d189e5-143c-4698-8118-11fcd3daa98e" containerID="fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3" exitCode=0
Sep 12 00:46:14.303410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.303270    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"
Sep 12 00:46:14.303410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.303310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" event={"ID":"a8d189e5-143c-4698-8118-11fcd3daa98e","Type":"ContainerDied","Data":"fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3"}
Sep 12 00:46:14.303410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.303390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm" event={"ID":"a8d189e5-143c-4698-8118-11fcd3daa98e","Type":"ContainerDied","Data":"1b06c47eb3db2b3da9557f9b24f2ce894ed7c95aefe26850e29e834778367d01"}
Sep 12 00:46:14.303599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.303413    2562 scope.go:117] "RemoveContainer" containerID="fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3"
Sep 12 00:46:14.313116 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.313103    2562 scope.go:117] "RemoveContainer" containerID="fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3"
Sep 12 00:46:14.313380 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:46:14.313361    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3\": container with ID starting with fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3 not found: ID does not exist" containerID="fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3"
Sep 12 00:46:14.313429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.313387    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3"} err="failed to get container status \"fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3\": rpc error: code = NotFound desc = could not find container \"fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3\": container with ID starting with fd75bce65ecc575032a3de1e56761866680825546d014e5ecf83d59197b194b3 not found: ID does not exist"
Sep 12 00:46:14.323925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.323902    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"]
Sep 12 00:46:14.326141 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.326121    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f12ab7-754d59b99d-vppsm"]
Sep 12 00:46:14.947586 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:14.947552    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:15.571272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:15.571236    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e469902-fc5d-4f45-9b1b-4497452d3eca" path="/var/lib/kubelet/pods/7e469902-fc5d-4f45-9b1b-4497452d3eca/volumes"
Sep 12 00:46:15.571756 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:15.571733    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d189e5-143c-4698-8118-11fcd3daa98e" path="/var/lib/kubelet/pods/a8d189e5-143c-4698-8118-11fcd3daa98e/volumes"
Sep 12 00:46:15.589133 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:15.589106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:17.299983 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:17.299950    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:46:19.853948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:19.853910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:19.917156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:19.917118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:20.125004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:20.124906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:27.226303 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226267    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"]
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226652    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d189e5-143c-4698-8118-11fcd3daa98e" containerName="maas-api"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226665    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d189e5-143c-4698-8118-11fcd3daa98e" containerName="maas-api"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226680    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40490a82-656f-423d-9f14-8969e5822ef0" containerName="main"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226687    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="40490a82-656f-423d-9f14-8969e5822ef0" containerName="main"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226707    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a44ad56-46ca-48cc-a49e-9ab749aec197" containerName="main"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226713    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a44ad56-46ca-48cc-a49e-9ab749aec197" containerName="main"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226721    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e469902-fc5d-4f45-9b1b-4497452d3eca" containerName="revoke-keys"
Sep 12 00:46:27.226763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226727    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e469902-fc5d-4f45-9b1b-4497452d3eca" containerName="revoke-keys"
Sep 12 00:46:27.227148 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226790    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e469902-fc5d-4f45-9b1b-4497452d3eca" containerName="revoke-keys"
Sep 12 00:46:27.227148 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226799    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="40490a82-656f-423d-9f14-8969e5822ef0" containerName="main"
Sep 12 00:46:27.227148 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226807    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d189e5-143c-4698-8118-11fcd3daa98e" containerName="maas-api"
Sep 12 00:46:27.227148 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.226814    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a44ad56-46ca-48cc-a49e-9ab749aec197" containerName="main"
Sep 12 00:46:27.231193 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.231174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.233547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.233524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"sub-test-model-e88680-kserve-self-signed-certs\""
Sep 12 00:46:27.234600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.234580    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"kube-root-ca.crt\""
Sep 12 00:46:27.234725 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.234618    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"openshift-service-ca.crt\""
Sep 12 00:46:27.234825 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.234803    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"default-dockercfg-bxwk5\""
Sep 12 00:46:27.237369 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.237347    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"]
Sep 12 00:46:27.262532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.262509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:27.300199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.300175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-model-cache\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.300294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.300206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tmp-dir\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.300294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.300224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-home\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.300416 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.300294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-dshm\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.300416 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.300322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sttj\" (UniqueName: \"kubernetes.io/projected/9e2bb392-341f-4c4c-a07c-13990f72e1b6-kube-api-access-8sttj\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.300416 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.300354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tls-certs\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401274 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-model-cache\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tmp-dir\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-home\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-dshm\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sttj\" (UniqueName: \"kubernetes.io/projected/9e2bb392-341f-4c4c-a07c-13990f72e1b6-kube-api-access-8sttj\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tls-certs\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-model-cache\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401717    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-home\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.401769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.401751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tmp-dir\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.403629 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.403603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-dshm\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.404024 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.404003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tls-certs\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.410085 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.410062    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sttj\" (UniqueName: \"kubernetes.io/projected/9e2bb392-341f-4c4c-a07c-13990f72e1b6-kube-api-access-8sttj\") pod \"sub-test-model-e88680-kserve-656f746dbf-hb8sl\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") " pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.541630 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.541558    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:27.809411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:27.809377    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"]
Sep 12 00:46:27.810972 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:46:27.810943    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e2bb392_341f_4c4c_a07c_13990f72e1b6.slice/crio-14639b70560c6adf50a72c5c7b990c86a424949ce9aa5ae0d55814df43438c55 WatchSource:0}: Error finding container 14639b70560c6adf50a72c5c7b990c86a424949ce9aa5ae0d55814df43438c55: Status 404 returned error can't find the container with id 14639b70560c6adf50a72c5c7b990c86a424949ce9aa5ae0d55814df43438c55
Sep 12 00:46:28.380002 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:28.379962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl" event={"ID":"9e2bb392-341f-4c4c-a07c-13990f72e1b6","Type":"ContainerStarted","Data":"e741478acb02c9909493853388b82dd378f7aefb18cb1b6aa4d9ef262d708903"}
Sep 12 00:46:28.380369 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:28.380021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl" event={"ID":"9e2bb392-341f-4c4c-a07c-13990f72e1b6","Type":"ContainerStarted","Data":"14639b70560c6adf50a72c5c7b990c86a424949ce9aa5ae0d55814df43438c55"}
Sep 12 00:46:28.396408 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:28.396344    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl" podStartSLOduration=1.39633047 podStartE2EDuration="1.39633047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:28.395207469 +0000 UTC m=+1791.332514001" watchObservedRunningTime="2026-09-12 00:46:28.39633047 +0000 UTC m=+1791.333636990"
Sep 12 00:46:29.958488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:29.958395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:46:30.110253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:30.110217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:46:37.542292 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:37.542253    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:37.542695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:37.542312    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:37.554849 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:37.554815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:46:37.607732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:37.607703    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:46:37.617411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:37.617381    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:46:38.447335 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:46:38.447299    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:47:04.309344 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:04.309303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:12.133559 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.133525    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"]
Sep 12 00:47:12.136921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.136902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.141079 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.141045    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"kube-root-ca.crt\""
Sep 12 00:47:12.141079 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.141076    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"sub-test-model-e37bfb-kserve-self-signed-certs\""
Sep 12 00:47:12.141269 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.141109    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"openshift-service-ca.crt\""
Sep 12 00:47:12.141269 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.141115    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"default-dockercfg-p2j96\""
Sep 12 00:47:12.155699 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.155675    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"]
Sep 12 00:47:12.161322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.161299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:12.214955 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.214922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-dshm\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.214955 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.214958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-home\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.215156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.215000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tmp-dir\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.215156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.215056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-model-cache\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.215156 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.215082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tls-certs\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.215262 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.215152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf55j\" (UniqueName: \"kubernetes.io/projected/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-kube-api-access-wf55j\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.315847 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.315809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-model-cache\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.315999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.315854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tls-certs\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.315999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.315890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf55j\" (UniqueName: \"kubernetes.io/projected/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-kube-api-access-wf55j\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.315999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.315909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-dshm\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.315999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.315925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-home\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.316213 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.315999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tmp-dir\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.316322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.316286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-model-cache\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.316499 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.316297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-home\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.316499 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.316406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tmp-dir\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.318288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.318266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-dshm\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.318382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.318367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tls-certs\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.325570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.325552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf55j\" (UniqueName: \"kubernetes.io/projected/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-kube-api-access-wf55j\") pod \"sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.446766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.446678    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:12.718780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:12.718755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"]
Sep 12 00:47:12.720786 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:47:12.720757    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d9fa96b_c8b6_475a_9b18_586fe0e91eb6.slice/crio-6c129df394a4c7f4e57ed21f666b63c96f6423b174cab9ce892ac1ca0c0cc5b0 WatchSource:0}: Error finding container 6c129df394a4c7f4e57ed21f666b63c96f6423b174cab9ce892ac1ca0c0cc5b0: Status 404 returned error can't find the container with id 6c129df394a4c7f4e57ed21f666b63c96f6423b174cab9ce892ac1ca0c0cc5b0
Sep 12 00:47:13.615728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:13.615689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld" event={"ID":"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6","Type":"ContainerStarted","Data":"a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0"}
Sep 12 00:47:13.615728 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:13.615731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld" event={"ID":"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6","Type":"ContainerStarted","Data":"6c129df394a4c7f4e57ed21f666b63c96f6423b174cab9ce892ac1ca0c0cc5b0"}
Sep 12 00:47:13.633613 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:13.633565    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld" podStartSLOduration=1.633542464 podStartE2EDuration="1.633542464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:47:13.632453214 +0000 UTC m=+1836.569759734" watchObservedRunningTime="2026-09-12 00:47:13.633542464 +0000 UTC m=+1836.570849018"
Sep 12 00:47:16.205707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:16.205666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:47:22.446914 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:22.446875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:22.447353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:22.446930    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:22.459548 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:22.459523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:22.675132 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:22.675104    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:47:30.030905 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:30.030861    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:47:33.568185 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:33.568150    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:39.403243 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:39.403206    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:47:47.720522 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:47.720464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"]
Sep 12 00:47:47.744289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:47.744262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:48.603992 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:48.603953    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"]
Sep 12 00:47:48.631258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:48.631222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:51.906300 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:51.906256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:47:53.153833 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.153799    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"]
Sep 12 00:47:53.158859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.158831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.162578 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.162554    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e88680\"/\"rate-test-model-e88680-kserve-self-signed-certs\""
Sep 12 00:47:53.172182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.172156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"]
Sep 12 00:47:53.226550 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.226515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:47:53.275483 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.275450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-tmp-dir\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.275618 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.275567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsmz\" (UniqueName: \"kubernetes.io/projected/30150f5a-1a98-41bb-b490-8820e985204d-kube-api-access-rnsmz\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.275618 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.275599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-home\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.275695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.275626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-dshm\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.275695 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.275674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-model-cache\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.275771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.275714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30150f5a-1a98-41bb-b490-8820e985204d-tls-certs\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.376999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.376961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsmz\" (UniqueName: \"kubernetes.io/projected/30150f5a-1a98-41bb-b490-8820e985204d-kube-api-access-rnsmz\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.376999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-home\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-dshm\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-model-cache\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377288 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30150f5a-1a98-41bb-b490-8820e985204d-tls-certs\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-tmp-dir\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377486 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-home\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-model-cache\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.377737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.377673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-tmp-dir\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.379380 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.379354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-dshm\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.379820 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.379802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30150f5a-1a98-41bb-b490-8820e985204d-tls-certs\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.386301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.386278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsmz\" (UniqueName: \"kubernetes.io/projected/30150f5a-1a98-41bb-b490-8820e985204d-kube-api-access-rnsmz\") pod \"rate-test-model-e88680-kserve-6994f5cccf-69c4c\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") " pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.469776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.469693    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:47:53.738929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.738905    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"]
Sep 12 00:47:53.740850 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:47:53.740820    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30150f5a_1a98_41bb_b490_8820e985204d.slice/crio-9997ca5a77e55e58bb501fb91cc62d6b439362bc946bd67e64ceccfb897769c7 WatchSource:0}: Error finding container 9997ca5a77e55e58bb501fb91cc62d6b439362bc946bd67e64ceccfb897769c7: Status 404 returned error can't find the container with id 9997ca5a77e55e58bb501fb91cc62d6b439362bc946bd67e64ceccfb897769c7
Sep 12 00:47:53.818015 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.817983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c" event={"ID":"30150f5a-1a98-41bb-b490-8820e985204d","Type":"ContainerStarted","Data":"d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb"}
Sep 12 00:47:53.818284 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.818029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c" event={"ID":"30150f5a-1a98-41bb-b490-8820e985204d","Type":"ContainerStarted","Data":"9997ca5a77e55e58bb501fb91cc62d6b439362bc946bd67e64ceccfb897769c7"}
Sep 12 00:47:53.837567 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:47:53.837523    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c" podStartSLOduration=0.837507027 podStartE2EDuration="837.507027ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:47:53.836423152 +0000 UTC m=+1876.773729671" watchObservedRunningTime="2026-09-12 00:47:53.837507027 +0000 UTC m=+1876.774813546"
Sep 12 00:48:02.108779 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:02.108699    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:48:02.741094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:02.741043    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl" podUID="9e2bb392-341f-4c4c-a07c-13990f72e1b6" containerName="main" containerID="cri-o://e741478acb02c9909493853388b82dd378f7aefb18cb1b6aa4d9ef262d708903" gracePeriod=45
Sep 12 00:48:02.867532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:02.867500    2562 generic.go:356] "Generic (PLEG): container finished" podID="9e2bb392-341f-4c4c-a07c-13990f72e1b6" containerID="e741478acb02c9909493853388b82dd378f7aefb18cb1b6aa4d9ef262d708903" exitCode=0
Sep 12 00:48:02.867665 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:02.867577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl" event={"ID":"9e2bb392-341f-4c4c-a07c-13990f72e1b6","Type":"ContainerDied","Data":"e741478acb02c9909493853388b82dd378f7aefb18cb1b6aa4d9ef262d708903"}
Sep 12 00:48:03.003532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.003460    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:48:03.064385 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064354    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-model-cache\") pod \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") "
Sep 12 00:48:03.064385 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064390    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tmp-dir\") pod \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") "
Sep 12 00:48:03.064658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-home\") pod \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") "
Sep 12 00:48:03.064658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064507    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sttj\" (UniqueName: \"kubernetes.io/projected/9e2bb392-341f-4c4c-a07c-13990f72e1b6-kube-api-access-8sttj\") pod \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") "
Sep 12 00:48:03.064658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064553    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-dshm\") pod \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") "
Sep 12 00:48:03.064658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tls-certs\") pod \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\" (UID: \"9e2bb392-341f-4c4c-a07c-13990f72e1b6\") "
Sep 12 00:48:03.064843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064696    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e2bb392-341f-4c4c-a07c-13990f72e1b6" (UID: "9e2bb392-341f-4c4c-a07c-13990f72e1b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.064843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064712    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-home" (OuterVolumeSpecName: "home") pod "9e2bb392-341f-4c4c-a07c-13990f72e1b6" (UID: "9e2bb392-341f-4c4c-a07c-13990f72e1b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.064843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e2bb392-341f-4c4c-a07c-13990f72e1b6" (UID: "9e2bb392-341f-4c4c-a07c-13990f72e1b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.065014 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.064995    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.065078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.065018    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.065078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.065028    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.066741 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.066716    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2bb392-341f-4c4c-a07c-13990f72e1b6-kube-api-access-8sttj" (OuterVolumeSpecName: "kube-api-access-8sttj") pod "9e2bb392-341f-4c4c-a07c-13990f72e1b6" (UID: "9e2bb392-341f-4c4c-a07c-13990f72e1b6"). InnerVolumeSpecName "kube-api-access-8sttj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:48:03.067112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.067094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e2bb392-341f-4c4c-a07c-13990f72e1b6" (UID: "9e2bb392-341f-4c4c-a07c-13990f72e1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:48:03.067166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.067117    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-dshm" (OuterVolumeSpecName: "dshm") pod "9e2bb392-341f-4c4c-a07c-13990f72e1b6" (UID: "9e2bb392-341f-4c4c-a07c-13990f72e1b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.166142 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.166106    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e2bb392-341f-4c4c-a07c-13990f72e1b6-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.166142 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.166138    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e2bb392-341f-4c4c-a07c-13990f72e1b6-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.166142 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.166148    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sttj\" (UniqueName: \"kubernetes.io/projected/9e2bb392-341f-4c4c-a07c-13990f72e1b6-kube-api-access-8sttj\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.470279 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.470238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:48:03.470461 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.470323    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:48:03.482940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.482914    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:48:03.624677 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.624616    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld" podUID="7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" containerName="main" containerID="cri-o://a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0" gracePeriod=45
Sep 12 00:48:03.875190 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.875160    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:48:03.875420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.875400    2562 generic.go:356] "Generic (PLEG): container finished" podID="7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" containerID="a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0" exitCode=0
Sep 12 00:48:03.875504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.875466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld" event={"ID":"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6","Type":"ContainerDied","Data":"a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0"}
Sep 12 00:48:03.875543 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.875529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld" event={"ID":"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6","Type":"ContainerDied","Data":"6c129df394a4c7f4e57ed21f666b63c96f6423b174cab9ce892ac1ca0c0cc5b0"}
Sep 12 00:48:03.875591 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.875548    2562 scope.go:117] "RemoveContainer" containerID="a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0"
Sep 12 00:48:03.876961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.876936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl" event={"ID":"9e2bb392-341f-4c4c-a07c-13990f72e1b6","Type":"ContainerDied","Data":"14639b70560c6adf50a72c5c7b990c86a424949ce9aa5ae0d55814df43438c55"}
Sep 12 00:48:03.876961 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.876958    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"
Sep 12 00:48:03.884888 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.884856    2562 scope.go:117] "RemoveContainer" containerID="a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0"
Sep 12 00:48:03.885238 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:48:03.885216    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0\": container with ID starting with a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0 not found: ID does not exist" containerID="a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0"
Sep 12 00:48:03.885327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.885244    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0"} err="failed to get container status \"a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0\": rpc error: code = NotFound desc = could not find container \"a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0\": container with ID starting with a49456b11a21432438665b55da9a25aa22b3a787dff2c8f8505c5ba24f7f44f0 not found: ID does not exist"
Sep 12 00:48:03.885327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.885260    2562 scope.go:117] "RemoveContainer" containerID="e741478acb02c9909493853388b82dd378f7aefb18cb1b6aa4d9ef262d708903"
Sep 12 00:48:03.889948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.889927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:48:03.926610 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.926580    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"]
Sep 12 00:48:03.929424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.929398    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/sub-test-model-e88680-kserve-656f746dbf-hb8sl"]
Sep 12 00:48:03.972684 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tmp-dir\") pod \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") "
Sep 12 00:48:03.972859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972731    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf55j\" (UniqueName: \"kubernetes.io/projected/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-kube-api-access-wf55j\") pod \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") "
Sep 12 00:48:03.972859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-dshm\") pod \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") "
Sep 12 00:48:03.972859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tls-certs\") pod \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") "
Sep 12 00:48:03.972859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-home\") pod \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") "
Sep 12 00:48:03.972859 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-model-cache\") pod \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\" (UID: \"7d9fa96b-c8b6-475a-9b18-586fe0e91eb6\") "
Sep 12 00:48:03.973136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.972959    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" (UID: "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.973136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.973092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-home" (OuterVolumeSpecName: "home") pod "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" (UID: "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.973242 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.973190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" (UID: "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:03.973301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.973272    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.973301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.973286    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.973301 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.973296    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:03.974930 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.974907    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" (UID: "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:48:03.975338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.975322    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-kube-api-access-wf55j" (OuterVolumeSpecName: "kube-api-access-wf55j") pod "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" (UID: "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6"). InnerVolumeSpecName "kube-api-access-wf55j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:48:03.975393 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:03.975325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-dshm" (OuterVolumeSpecName: "dshm") pod "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" (UID: "7d9fa96b-c8b6-475a-9b18-586fe0e91eb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:48:04.074774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:04.074737    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:04.074774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:04.074769    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf55j\" (UniqueName: \"kubernetes.io/projected/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-kube-api-access-wf55j\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:04.074774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:04.074779    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:48:04.884560 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:04.884528    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"
Sep 12 00:48:04.905140 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:04.905117    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"]
Sep 12 00:48:04.908638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:04.908613    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/sub-test-model-e37bfb-kserve-5fb7fc66b9-6m6ld"]
Sep 12 00:48:05.570662 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:05.570627    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" path="/var/lib/kubelet/pods/7d9fa96b-c8b6-475a-9b18-586fe0e91eb6/volumes"
Sep 12 00:48:05.571047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:05.571033    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2bb392-341f-4c4c-a07c-13990f72e1b6" path="/var/lib/kubelet/pods/9e2bb392-341f-4c4c-a07c-13990f72e1b6/volumes"
Sep 12 00:48:08.494893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:08.494858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:12.087858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:12.087818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:13.235258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.235223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:13.329280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.329245    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"]
Sep 12 00:48:13.329839 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.329822    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" containerName="main"
Sep 12 00:48:13.329889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.329843    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" containerName="main"
Sep 12 00:48:13.329889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.329872    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2bb392-341f-4c4c-a07c-13990f72e1b6" containerName="main"
Sep 12 00:48:13.329889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.329882    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2bb392-341f-4c4c-a07c-13990f72e1b6" containerName="main"
Sep 12 00:48:13.329995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.329983    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2bb392-341f-4c4c-a07c-13990f72e1b6" containerName="main"
Sep 12 00:48:13.330034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.330000    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9fa96b-c8b6-475a-9b18-586fe0e91eb6" containerName="main"
Sep 12 00:48:13.333258 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.333233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.336862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.336806    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"rate-test-model-e37bfb-kserve-self-signed-certs\""
Sep 12 00:48:13.336993 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.336872    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"openshift-service-ca.crt\""
Sep 12 00:48:13.336993 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.336984    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"default-dockercfg-p2j96\""
Sep 12 00:48:13.337274 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.337213    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e37bfb\"/\"kube-root-ca.crt\""
Sep 12 00:48:13.340244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.340225    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"]
Sep 12 00:48:13.366285 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.366262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:13.456135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.456105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-dshm\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.456314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.456145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-model-cache\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.456314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.456253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-home\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.456314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.456290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-tmp-dir\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.456466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.456325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jc8m\" (UniqueName: \"kubernetes.io/projected/45dfcb54-4ad1-44ad-b455-ac67499de880-kube-api-access-7jc8m\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.456466 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.456353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dfcb54-4ad1-44ad-b455-ac67499de880-tls-certs\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.556967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-home\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-tmp-dir\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jc8m\" (UniqueName: \"kubernetes.io/projected/45dfcb54-4ad1-44ad-b455-ac67499de880-kube-api-access-7jc8m\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dfcb54-4ad1-44ad-b455-ac67499de880-tls-certs\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-dshm\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-model-cache\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-home\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557539 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-tmp-dir\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.557610 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.557566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-model-cache\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.559415 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.559390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-dshm\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.559795 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.559773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dfcb54-4ad1-44ad-b455-ac67499de880-tls-certs\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.565461 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.565439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jc8m\" (UniqueName: \"kubernetes.io/projected/45dfcb54-4ad1-44ad-b455-ac67499de880-kube-api-access-7jc8m\") pod \"rate-test-model-e37bfb-kserve-556f4575ff-jkllc\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") " pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.644391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.644364    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:13.914854 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.914822    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"]
Sep 12 00:48:13.918526 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:48:13.918493    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45dfcb54_4ad1_44ad_b455_ac67499de880.slice/crio-57856c1dc1b1c4fb4e9cceb18daf3cd1e579d3f841e268a5681acaed7bf471bc WatchSource:0}: Error finding container 57856c1dc1b1c4fb4e9cceb18daf3cd1e579d3f841e268a5681acaed7bf471bc: Status 404 returned error can't find the container with id 57856c1dc1b1c4fb4e9cceb18daf3cd1e579d3f841e268a5681acaed7bf471bc
Sep 12 00:48:13.934902 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:13.934869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc" event={"ID":"45dfcb54-4ad1-44ad-b455-ac67499de880","Type":"ContainerStarted","Data":"57856c1dc1b1c4fb4e9cceb18daf3cd1e579d3f841e268a5681acaed7bf471bc"}
Sep 12 00:48:14.941280 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:14.941239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc" event={"ID":"45dfcb54-4ad1-44ad-b455-ac67499de880","Type":"ContainerStarted","Data":"bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e"}
Sep 12 00:48:14.957337 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:14.957286    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc" podStartSLOduration=1.957271729 podStartE2EDuration="1.957271729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:14.956851164 +0000 UTC m=+1897.894157684" watchObservedRunningTime="2026-09-12 00:48:14.957271729 +0000 UTC m=+1897.894578249"
Sep 12 00:48:17.108402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:17.108352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:48:23.644870 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:23.644829    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:23.645296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:23.644882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:23.658363 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:23.658331    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:24.002071 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:24.001995    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:48:36.503412 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:36.503368    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:48:45.702781 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:45.702744    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:48:50.583093 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:50.583058    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"]
Sep 12 00:48:50.604167 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:50.604140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:51.623324 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:51.623291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"]
Sep 12 00:48:51.645046 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:51.645018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:52.094313 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:52.094286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:52.416883 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:52.416802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:52.649283 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:52.649250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:54.899503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:54.899456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:48:55.128766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:55.128726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:55.230742 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:55.230659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:55.765514 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:55.765459    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:55.861493 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:55.861437    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:55.958652 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:55.958615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:56.853529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:56.853497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:57.575189 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.575157    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"]
Sep 12 00:48:57.578551 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.578533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.581033 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.581011    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 00:48:57.582147 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.582118    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 00:48:57.582254 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.582157    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f4scw\""
Sep 12 00:48:57.582254 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.582186    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-64e462-serving-cert\""
Sep 12 00:48:57.582254 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.582189    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:48:57.589234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.589210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"]
Sep 12 00:48:57.627516 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.627465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:57.657387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.657356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efce3f42-da26-46cb-88dd-5b697f7c62ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.657599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.657518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-metrics-certs\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.657599 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.657553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.657721 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.657682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49mm\" (UniqueName: \"kubernetes.io/projected/efce3f42-da26-46cb-88dd-5b697f7c62ae-kube-api-access-n49mm\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.728536 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.728502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:57.758590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.758559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-metrics-certs\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.758715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.758593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.758715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.758663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49mm\" (UniqueName: \"kubernetes.io/projected/efce3f42-da26-46cb-88dd-5b697f7c62ae-kube-api-access-n49mm\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.758715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.758697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efce3f42-da26-46cb-88dd-5b697f7c62ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.758883 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:48:57.758743    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-serving-cert: secret "maas-api-e2e-ipp-a-64e462-serving-cert" not found
Sep 12 00:48:57.758883 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:48:57.758806    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls podName:efce3f42-da26-46cb-88dd-5b697f7c62ae nodeName:}" failed. No retries permitted until 2026-09-12 00:48:58.258789423 +0000 UTC m=+1941.196095923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls") pod "maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" (UID: "efce3f42-da26-46cb-88dd-5b697f7c62ae") : secret "maas-api-e2e-ipp-a-64e462-serving-cert" not found
Sep 12 00:48:57.759340 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.759319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efce3f42-da26-46cb-88dd-5b697f7c62ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.760999 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.760979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-metrics-certs\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.769520 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.769500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49mm\" (UniqueName: \"kubernetes.io/projected/efce3f42-da26-46cb-88dd-5b697f7c62ae-kube-api-access-n49mm\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:57.840402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:57.840332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:58.100129 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.100053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:58.201217 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.201181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:48:58.202248 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.202228    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"]
Sep 12 00:48:58.205635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.205620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.208134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.208114    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-64e462\""
Sep 12 00:48:58.208263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.208189    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 00:48:58.208263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.208244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-64e462-dockercfg-fvmxn\""
Sep 12 00:48:58.208397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.208260    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 00:48:58.216081 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.216058    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"]
Sep 12 00:48:58.263533 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.263507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:58.266116 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.266095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:58.364892 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.364824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhh9z\" (UniqueName: \"kubernetes.io/projected/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-kube-api-access-nhh9z\") pod \"payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.365006 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.364922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.466220 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.466186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.466370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.466240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh9z\" (UniqueName: \"kubernetes.io/projected/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-kube-api-access-nhh9z\") pod \"payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.467168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.467142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.477053 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.477031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh9z\" (UniqueName: \"kubernetes.io/projected/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-kube-api-access-nhh9z\") pod \"payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.489945 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.489921    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:48:58.517131 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.517105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:58.760584 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.760557    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"]
Sep 12 00:48:58.761655 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:48:58.761625    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefce3f42_da26_46cb_88dd_5b697f7c62ae.slice/crio-53f51cd5e08ac201825331fe630035bf8658ed36d824b7ec919898a5ec1a7217 WatchSource:0}: Error finding container 53f51cd5e08ac201825331fe630035bf8658ed36d824b7ec919898a5ec1a7217: Status 404 returned error can't find the container with id 53f51cd5e08ac201825331fe630035bf8658ed36d824b7ec919898a5ec1a7217
Sep 12 00:48:58.763236 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.763214    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:48:58.797423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:58.797398    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"]
Sep 12 00:48:58.798771 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:48:58.798748    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4417e401_c8cc_4522_88a3_d2c42c1a4bfe.slice/crio-56d0e3f4e2ae56e5d163eb39eb1368e947a735b2d3dae49fced62d495d353ad8 WatchSource:0}: Error finding container 56d0e3f4e2ae56e5d163eb39eb1368e947a735b2d3dae49fced62d495d353ad8: Status 404 returned error can't find the container with id 56d0e3f4e2ae56e5d163eb39eb1368e947a735b2d3dae49fced62d495d353ad8
Sep 12 00:48:59.177147 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:59.177106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx" event={"ID":"4417e401-c8cc-4522-88a3-d2c42c1a4bfe","Type":"ContainerStarted","Data":"7489b31cab9c2e9128fcb45951fcbc42762c6b1c7d5eae3827280befe421defb"}
Sep 12 00:48:59.177333 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:59.177182    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:48:59.177333 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:59.177208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx" event={"ID":"4417e401-c8cc-4522-88a3-d2c42c1a4bfe","Type":"ContainerStarted","Data":"56d0e3f4e2ae56e5d163eb39eb1368e947a735b2d3dae49fced62d495d353ad8"}
Sep 12 00:48:59.178424 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:59.178399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" event={"ID":"efce3f42-da26-46cb-88dd-5b697f7c62ae","Type":"ContainerStarted","Data":"53f51cd5e08ac201825331fe630035bf8658ed36d824b7ec919898a5ec1a7217"}
Sep 12 00:48:59.194069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:48:59.194023    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx" podStartSLOduration=1.194009289 podStartE2EDuration="1.194009289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:59.19159749 +0000 UTC m=+1942.128904024" watchObservedRunningTime="2026-09-12 00:48:59.194009289 +0000 UTC m=+1942.131315809"
Sep 12 00:49:00.185394 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:00.185355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" event={"ID":"efce3f42-da26-46cb-88dd-5b697f7c62ae","Type":"ContainerStarted","Data":"25060d8280c79d46fb77fef152566eb03db7d5082ef1479e824ef6371b6454ca"}
Sep 12 00:49:00.185858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:00.185649    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:49:00.202295 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:00.202248    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" podStartSLOduration=2.744947879 podStartE2EDuration="3.202234037s" totalImagesPullingTime="457.286158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:00.200995425 +0000 UTC m=+1943.138301948" watchObservedRunningTime="2026-09-12 00:49:00.202234037 +0000 UTC m=+1943.139540557"
Sep 12 00:49:05.601882 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.601826    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c" podUID="30150f5a-1a98-41bb-b490-8820e985204d" containerName="main" containerID="cri-o://d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb" gracePeriod=45
Sep 12 00:49:05.675140 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.675089    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:05.774422 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.774389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:05.850060 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.850033    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:49:05.935688 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935605    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-model-cache\") pod \"30150f5a-1a98-41bb-b490-8820e985204d\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") "
Sep 12 00:49:05.935688 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935652    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-dshm\") pod \"30150f5a-1a98-41bb-b490-8820e985204d\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") "
Sep 12 00:49:05.935889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnsmz\" (UniqueName: \"kubernetes.io/projected/30150f5a-1a98-41bb-b490-8820e985204d-kube-api-access-rnsmz\") pod \"30150f5a-1a98-41bb-b490-8820e985204d\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") "
Sep 12 00:49:05.935889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935795    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30150f5a-1a98-41bb-b490-8820e985204d-tls-certs\") pod \"30150f5a-1a98-41bb-b490-8820e985204d\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") "
Sep 12 00:49:05.935889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935850    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-tmp-dir\") pod \"30150f5a-1a98-41bb-b490-8820e985204d\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") "
Sep 12 00:49:05.935889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-model-cache" (OuterVolumeSpecName: "model-cache") pod "30150f5a-1a98-41bb-b490-8820e985204d" (UID: "30150f5a-1a98-41bb-b490-8820e985204d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:05.935889 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.935877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-home\") pod \"30150f5a-1a98-41bb-b490-8820e985204d\" (UID: \"30150f5a-1a98-41bb-b490-8820e985204d\") "
Sep 12 00:49:05.936160 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.936063    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30150f5a-1a98-41bb-b490-8820e985204d" (UID: "30150f5a-1a98-41bb-b490-8820e985204d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:05.936226 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.936152    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-home" (OuterVolumeSpecName: "home") pod "30150f5a-1a98-41bb-b490-8820e985204d" (UID: "30150f5a-1a98-41bb-b490-8820e985204d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:05.936287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.936225    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:05.936287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.936238    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:05.937846 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.937821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-dshm" (OuterVolumeSpecName: "dshm") pod "30150f5a-1a98-41bb-b490-8820e985204d" (UID: "30150f5a-1a98-41bb-b490-8820e985204d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:05.937995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.937893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30150f5a-1a98-41bb-b490-8820e985204d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30150f5a-1a98-41bb-b490-8820e985204d" (UID: "30150f5a-1a98-41bb-b490-8820e985204d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:49:05.937995 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:05.937962    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30150f5a-1a98-41bb-b490-8820e985204d-kube-api-access-rnsmz" (OuterVolumeSpecName: "kube-api-access-rnsmz") pod "30150f5a-1a98-41bb-b490-8820e985204d" (UID: "30150f5a-1a98-41bb-b490-8820e985204d"). InnerVolumeSpecName "kube-api-access-rnsmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:06.037732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.037699    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:06.037732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.037728    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsmz\" (UniqueName: \"kubernetes.io/projected/30150f5a-1a98-41bb-b490-8820e985204d-kube-api-access-rnsmz\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:06.037732 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.037739    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30150f5a-1a98-41bb-b490-8820e985204d-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:06.037936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.037748    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30150f5a-1a98-41bb-b490-8820e985204d-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:06.197576 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.197492    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:49:06.219751 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.219716    2562 generic.go:356] "Generic (PLEG): container finished" podID="30150f5a-1a98-41bb-b490-8820e985204d" containerID="d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb" exitCode=0
Sep 12 00:49:06.219904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.219757    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"
Sep 12 00:49:06.219904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.219796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c" event={"ID":"30150f5a-1a98-41bb-b490-8820e985204d","Type":"ContainerDied","Data":"d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb"}
Sep 12 00:49:06.219904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.219840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c" event={"ID":"30150f5a-1a98-41bb-b490-8820e985204d","Type":"ContainerDied","Data":"9997ca5a77e55e58bb501fb91cc62d6b439362bc946bd67e64ceccfb897769c7"}
Sep 12 00:49:06.219904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.219862    2562 scope.go:117] "RemoveContainer" containerID="d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb"
Sep 12 00:49:06.230553 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.230322    2562 scope.go:117] "RemoveContainer" containerID="d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb"
Sep 12 00:49:06.230737 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:49:06.230714    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb\": container with ID starting with d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb not found: ID does not exist" containerID="d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb"
Sep 12 00:49:06.230796 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.230745    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb"} err="failed to get container status \"d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb\": rpc error: code = NotFound desc = could not find container \"d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb\": container with ID starting with d26a29a689a8229d772a0fac266413217a2f6a666c879c9e09859a04ab0912fb not found: ID does not exist"
Sep 12 00:49:06.245658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.245623    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"]
Sep 12 00:49:06.249975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.249943    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e88680/rate-test-model-e88680-kserve-6994f5cccf-69c4c"]
Sep 12 00:49:06.642587 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.642548    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc" podUID="45dfcb54-4ad1-44ad-b455-ac67499de880" containerName="main" containerID="cri-o://bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e" gracePeriod=45
Sep 12 00:49:06.893393 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:06.893341    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:49:07.048814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.048781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jc8m\" (UniqueName: \"kubernetes.io/projected/45dfcb54-4ad1-44ad-b455-ac67499de880-kube-api-access-7jc8m\") pod \"45dfcb54-4ad1-44ad-b455-ac67499de880\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") "
Sep 12 00:49:07.049000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.048846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-model-cache\") pod \"45dfcb54-4ad1-44ad-b455-ac67499de880\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") "
Sep 12 00:49:07.049000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.048878    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dfcb54-4ad1-44ad-b455-ac67499de880-tls-certs\") pod \"45dfcb54-4ad1-44ad-b455-ac67499de880\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") "
Sep 12 00:49:07.049000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.048903    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-home\") pod \"45dfcb54-4ad1-44ad-b455-ac67499de880\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") "
Sep 12 00:49:07.049000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.048997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-tmp-dir\") pod \"45dfcb54-4ad1-44ad-b455-ac67499de880\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") "
Sep 12 00:49:07.049249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.049052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-dshm\") pod \"45dfcb54-4ad1-44ad-b455-ac67499de880\" (UID: \"45dfcb54-4ad1-44ad-b455-ac67499de880\") "
Sep 12 00:49:07.049249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.049193    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-model-cache" (OuterVolumeSpecName: "model-cache") pod "45dfcb54-4ad1-44ad-b455-ac67499de880" (UID: "45dfcb54-4ad1-44ad-b455-ac67499de880"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:07.049360 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.049286    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-home" (OuterVolumeSpecName: "home") pod "45dfcb54-4ad1-44ad-b455-ac67499de880" (UID: "45dfcb54-4ad1-44ad-b455-ac67499de880"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:07.049420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.049373    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45dfcb54-4ad1-44ad-b455-ac67499de880" (UID: "45dfcb54-4ad1-44ad-b455-ac67499de880"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:07.049496 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.049426    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:07.049496 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.049447    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:07.051177 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.051148    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dfcb54-4ad1-44ad-b455-ac67499de880-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45dfcb54-4ad1-44ad-b455-ac67499de880" (UID: "45dfcb54-4ad1-44ad-b455-ac67499de880"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:49:07.051689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.051658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dfcb54-4ad1-44ad-b455-ac67499de880-kube-api-access-7jc8m" (OuterVolumeSpecName: "kube-api-access-7jc8m") pod "45dfcb54-4ad1-44ad-b455-ac67499de880" (UID: "45dfcb54-4ad1-44ad-b455-ac67499de880"). InnerVolumeSpecName "kube-api-access-7jc8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:07.051752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.051662    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-dshm" (OuterVolumeSpecName: "dshm") pod "45dfcb54-4ad1-44ad-b455-ac67499de880" (UID: "45dfcb54-4ad1-44ad-b455-ac67499de880"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:07.150206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.150134    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dfcb54-4ad1-44ad-b455-ac67499de880-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:07.150206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.150164    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:07.150206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.150179    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dfcb54-4ad1-44ad-b455-ac67499de880-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:07.150206 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.150192    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jc8m\" (UniqueName: \"kubernetes.io/projected/45dfcb54-4ad1-44ad-b455-ac67499de880-kube-api-access-7jc8m\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:49:07.227595 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.227563    2562 generic.go:356] "Generic (PLEG): container finished" podID="45dfcb54-4ad1-44ad-b455-ac67499de880" containerID="bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e" exitCode=0
Sep 12 00:49:07.227715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.227608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"
Sep 12 00:49:07.227715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.227633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc" event={"ID":"45dfcb54-4ad1-44ad-b455-ac67499de880","Type":"ContainerDied","Data":"bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e"}
Sep 12 00:49:07.227715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.227666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc" event={"ID":"45dfcb54-4ad1-44ad-b455-ac67499de880","Type":"ContainerDied","Data":"57856c1dc1b1c4fb4e9cceb18daf3cd1e579d3f841e268a5681acaed7bf471bc"}
Sep 12 00:49:07.227715 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.227682    2562 scope.go:117] "RemoveContainer" containerID="bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e"
Sep 12 00:49:07.236646 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.236630    2562 scope.go:117] "RemoveContainer" containerID="bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e"
Sep 12 00:49:07.236911 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:49:07.236884    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e\": container with ID starting with bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e not found: ID does not exist" containerID="bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e"
Sep 12 00:49:07.236967 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.236913    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e"} err="failed to get container status \"bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e\": rpc error: code = NotFound desc = could not find container \"bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e\": container with ID starting with bfddf8264b19fd39701ea0a1a544fc075c7cd62aef066a3c83bf5a424737392e not found: ID does not exist"
Sep 12 00:49:07.247145 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.247121    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"]
Sep 12 00:49:07.250860 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.250838    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e37bfb/rate-test-model-e37bfb-kserve-556f4575ff-jkllc"]
Sep 12 00:49:07.571127 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.571093    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30150f5a-1a98-41bb-b490-8820e985204d" path="/var/lib/kubelet/pods/30150f5a-1a98-41bb-b490-8820e985204d/volumes"
Sep 12 00:49:07.571512 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.571498    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45dfcb54-4ad1-44ad-b455-ac67499de880" path="/var/lib/kubelet/pods/45dfcb54-4ad1-44ad-b455-ac67499de880/volumes"
Sep 12 00:49:07.805381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:07.805338    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:49:10.187338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:10.187310    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:49:19.563607 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:19.563566    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:20.334702 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:20.334667    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:20.864918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:20.864892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:21.034752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:21.034719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:21.515287 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:21.515251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:21.766915 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:21.766829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:21.858625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:21.858595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:22.025685 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:22.025611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:24.587063 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:24.587028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:24.928334 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:24.928209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:25.032240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:25.032205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:25.471090 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:25.471057    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:26.232263 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.232227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:26.855510 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.855462    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"]
Sep 12 00:49:26.855888 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.855867    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45dfcb54-4ad1-44ad-b455-ac67499de880" containerName="main"
Sep 12 00:49:26.856028 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.855910    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="45dfcb54-4ad1-44ad-b455-ac67499de880" containerName="main"
Sep 12 00:49:26.856028 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.855953    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30150f5a-1a98-41bb-b490-8820e985204d" containerName="main"
Sep 12 00:49:26.856028 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.855963    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30150f5a-1a98-41bb-b490-8820e985204d" containerName="main"
Sep 12 00:49:26.856188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.856080    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30150f5a-1a98-41bb-b490-8820e985204d" containerName="main"
Sep 12 00:49:26.856188 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.856098    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="45dfcb54-4ad1-44ad-b455-ac67499de880" containerName="main"
Sep 12 00:49:26.859157 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.859136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:26.861846 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.861822    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1cbe1c-serving-cert\""
Sep 12 00:49:26.868557 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.868485    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"]
Sep 12 00:49:26.891606 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.891581    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:26.932012 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.931983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:26.932137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.932021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:26.932137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.932060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:26.932137 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.932129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5n9\" (UniqueName: \"kubernetes.io/projected/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-kube-api-access-8c5n9\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:26.947400 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:26.947371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:27.032826 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.032793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5n9\" (UniqueName: \"kubernetes.io/projected/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-kube-api-access-8c5n9\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.033015 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.032840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.033015 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.032863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.033015 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.032889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.033571 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.033544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.035338 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.035306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.035426 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.035372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.042311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.042287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5n9\" (UniqueName: \"kubernetes.io/projected/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-kube-api-access-8c5n9\") pod \"maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.171370 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.171281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:27.350195 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.350163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:27.441649 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.441575    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"]
Sep 12 00:49:27.443921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.443885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:27.444913 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:49:27.444891    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897c93de_6e20_4c96_adc2_fc8d2aa3e2b2.slice/crio-2b74c6acbc2bec9488de8719028a00dbc8b934c06421a58da52cee3d9b8ae5ba WatchSource:0}: Error finding container 2b74c6acbc2bec9488de8719028a00dbc8b934c06421a58da52cee3d9b8ae5ba: Status 404 returned error can't find the container with id 2b74c6acbc2bec9488de8719028a00dbc8b934c06421a58da52cee3d9b8ae5ba
Sep 12 00:49:27.450696 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.450672    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"]
Sep 12 00:49:27.455068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.455051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.457627 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.457604    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1cbe1c\""
Sep 12 00:49:27.457716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.457638    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1cbe1c-dockercfg-ndd62\""
Sep 12 00:49:27.464183 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.464158    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"]
Sep 12 00:49:27.495968 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.495945    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"]
Sep 12 00:49:27.499421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.499404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.509314 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.509293    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"]
Sep 12 00:49:27.537431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.537405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.537553 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.537439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqb4p\" (UniqueName: \"kubernetes.io/projected/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-kube-api-access-zqb4p\") pod \"payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.638249 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.638220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98gn\" (UniqueName: \"kubernetes.io/projected/a386bdfa-89b5-4882-a387-467eacd29bda-kube-api-access-d98gn\") pod \"payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.638406 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.638369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a386bdfa-89b5-4882-a387-467eacd29bda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.638503 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.638415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.638580 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.638499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqb4p\" (UniqueName: \"kubernetes.io/projected/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-kube-api-access-zqb4p\") pod \"payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.639109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.639090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.646745 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.646724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqb4p\" (UniqueName: \"kubernetes.io/projected/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-kube-api-access-zqb4p\") pod \"payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.739935 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.739868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a386bdfa-89b5-4882-a387-467eacd29bda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.739935 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.739919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98gn\" (UniqueName: \"kubernetes.io/projected/a386bdfa-89b5-4882-a387-467eacd29bda-kube-api-access-d98gn\") pod \"payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.740533 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.740513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a386bdfa-89b5-4882-a387-467eacd29bda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.748040 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.748010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98gn\" (UniqueName: \"kubernetes.io/projected/a386bdfa-89b5-4882-a387-467eacd29bda-kube-api-access-d98gn\") pod \"payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:27.766425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.766398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:27.809402 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:27.809379    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:28.039025 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.039002    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"]
Sep 12 00:49:28.041362 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:49:28.041327    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ec62c0_ce47_4ad9_bc19_48e5a065ce57.slice/crio-3d199186fa55ad46f7988d97dec49f5dd804333ce85cf1d322ae57429f48af4d WatchSource:0}: Error finding container 3d199186fa55ad46f7988d97dec49f5dd804333ce85cf1d322ae57429f48af4d: Status 404 returned error can't find the container with id 3d199186fa55ad46f7988d97dec49f5dd804333ce85cf1d322ae57429f48af4d
Sep 12 00:49:28.095637 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.095612    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"]
Sep 12 00:49:28.096958 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:49:28.096938    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda386bdfa_89b5_4882_a387_467eacd29bda.slice/crio-e20d5ff6d59b223fd2b3f7ae072163fa02039c9a348890b9ecdf636a9c1a586f WatchSource:0}: Error finding container e20d5ff6d59b223fd2b3f7ae072163fa02039c9a348890b9ecdf636a9c1a586f: Status 404 returned error can't find the container with id e20d5ff6d59b223fd2b3f7ae072163fa02039c9a348890b9ecdf636a9c1a586f
Sep 12 00:49:28.340386 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.340351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd" event={"ID":"74ec62c0-ce47-4ad9-bc19-48e5a065ce57","Type":"ContainerStarted","Data":"a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630"}
Sep 12 00:49:28.340594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.340446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd" event={"ID":"74ec62c0-ce47-4ad9-bc19-48e5a065ce57","Type":"ContainerStarted","Data":"3d199186fa55ad46f7988d97dec49f5dd804333ce85cf1d322ae57429f48af4d"}
Sep 12 00:49:28.340594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.340499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:28.341917 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.341891    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598" event={"ID":"a386bdfa-89b5-4882-a387-467eacd29bda","Type":"ContainerStarted","Data":"44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42"}
Sep 12 00:49:28.342033 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.341944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598" event={"ID":"a386bdfa-89b5-4882-a387-467eacd29bda","Type":"ContainerStarted","Data":"e20d5ff6d59b223fd2b3f7ae072163fa02039c9a348890b9ecdf636a9c1a586f"}
Sep 12 00:49:28.342033 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.341972    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:28.343562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.343536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2" event={"ID":"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2","Type":"ContainerStarted","Data":"101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4"}
Sep 12 00:49:28.343650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.343585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2" event={"ID":"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2","Type":"ContainerStarted","Data":"2b74c6acbc2bec9488de8719028a00dbc8b934c06421a58da52cee3d9b8ae5ba"}
Sep 12 00:49:28.343650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.343618    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:28.358324 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.358272    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd" podStartSLOduration=1.35825321 podStartE2EDuration="1.35825321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:28.355294801 +0000 UTC m=+1971.292601323" watchObservedRunningTime="2026-09-12 00:49:28.35825321 +0000 UTC m=+1971.295559732"
Sep 12 00:49:28.372182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.372140    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2" podStartSLOduration=2.047823744 podStartE2EDuration="2.372122738s" totalImagesPullingTime="324.298994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:28.370853465 +0000 UTC m=+1971.308159984" watchObservedRunningTime="2026-09-12 00:49:28.372122738 +0000 UTC m=+1971.309429262"
Sep 12 00:49:28.388347 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:28.388303    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598" podStartSLOduration=1.3882902480000001 podStartE2EDuration="1.388290248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:28.386185659 +0000 UTC m=+1971.323492179" watchObservedRunningTime="2026-09-12 00:49:28.388290248 +0000 UTC m=+1971.325596782"
Sep 12 00:49:32.564578 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:32.564494    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:49:34.359903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:34.359839    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:49:39.356191 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:39.356146    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:49:39.356763 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:39.356240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:49:48.994094 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:48.994058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:49.730451 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:49.730424    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:49.958769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:49.958729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:50.463823 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:50.463783    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:51.135358 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:51.135323    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:51.947526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:51.947490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:49:55.234806 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:49:55.234772    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598_a386bdfa-89b5-4882-a387-467eacd29bda/payload-processing/0.log"
Sep 12 00:50:17.063694 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:17.063657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:18.476140 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:18.476084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:18.578278 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:18.578245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:20.810104 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:20.810067    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:50:36.724159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:36.724125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:40.114346 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:40.114305    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:50:46.821674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:46.821642    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598_a386bdfa-89b5-4882-a387-467eacd29bda/payload-processing/0.log"
Sep 12 00:50:47.101523 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:47.101427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:47.457462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:47.457374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:48.169046 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:48.169007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:50.244679 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:50.244639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:50.416225 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:50.416183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:50.514675 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:50.514589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:51.059234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:51.059172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:51.971012 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:51.970972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:52.620799 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:52.620764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:53.255703 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.255667    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"]
Sep 12 00:50:53.259521 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.259495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.262294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.262270    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b4b570-serving-cert\""
Sep 12 00:50:53.268839 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.268815    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"]
Sep 12 00:50:53.299401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.299363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:53.333918 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.333883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.334122 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.333930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.334122 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.334021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwsv\" (UniqueName: \"kubernetes.io/projected/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-kube-api-access-4pwsv\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.334234 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.334129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.399723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.399690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:53.435174 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.435139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.435310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.435185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.435310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.435234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwsv\" (UniqueName: \"kubernetes.io/projected/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-kube-api-access-4pwsv\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.435310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.435302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.435465 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:50:53.435449    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-serving-cert: secret "maas-api-e2e-ipp-cleanup-b4b570-serving-cert" not found
Sep 12 00:50:53.435601 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:50:53.435588    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls podName:9c1c7e94-dd51-47c5-a8b2-4837162db0f9 nodeName:}" failed. No retries permitted until 2026-09-12 00:50:53.935566434 +0000 UTC m=+2056.872872932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" (UID: "9c1c7e94-dd51-47c5-a8b2-4837162db0f9") : secret "maas-api-e2e-ipp-cleanup-b4b570-serving-cert" not found
Sep 12 00:50:53.435989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.435966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.437764 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.437739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.446515 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.446463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwsv\" (UniqueName: \"kubernetes.io/projected/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-kube-api-access-4pwsv\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.768845 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.768811    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:53.866382 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.866345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:53.904557 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.904522    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"]
Sep 12 00:50:53.908164 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.908147    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:53.910740 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.910715    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b4b570-dockercfg-lgspr\""
Sep 12 00:50:53.910873 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.910755    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b4b570\""
Sep 12 00:50:53.917760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.917734    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"]
Sep 12 00:50:53.941159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.941123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzwq\" (UniqueName: \"kubernetes.io/projected/23ff2668-da48-4a4a-a6ca-36747187b805-kube-api-access-7bzwq\") pod \"payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:53.941327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.941184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ff2668-da48-4a4a-a6ca-36747187b805-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:53.941327 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:53.941233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:53.941403 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:50:53.941346    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-serving-cert: secret "maas-api-e2e-ipp-cleanup-b4b570-serving-cert" not found
Sep 12 00:50:53.941438 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:50:53.941415    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls podName:9c1c7e94-dd51-47c5-a8b2-4837162db0f9 nodeName:}" failed. No retries permitted until 2026-09-12 00:50:54.94139991 +0000 UTC m=+2057.878706408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" (UID: "9c1c7e94-dd51-47c5-a8b2-4837162db0f9") : secret "maas-api-e2e-ipp-cleanup-b4b570-serving-cert" not found
Sep 12 00:50:54.032312 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.032218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:50:54.042457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.042416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzwq\" (UniqueName: \"kubernetes.io/projected/23ff2668-da48-4a4a-a6ca-36747187b805-kube-api-access-7bzwq\") pod \"payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:54.042633 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.042519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ff2668-da48-4a4a-a6ca-36747187b805-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:54.043222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.043202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ff2668-da48-4a4a-a6ca-36747187b805-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:54.051289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.051256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzwq\" (UniqueName: \"kubernetes.io/projected/23ff2668-da48-4a4a-a6ca-36747187b805-kube-api-access-7bzwq\") pod \"payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:54.221032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.220992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:54.492306 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.492280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"]
Sep 12 00:50:54.493716 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:50:54.493685    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ff2668_da48_4a4a_a6ca_36747187b805.slice/crio-78d3cae2616f02d8477bdcac62923194dd3a6cf193f97cbd2d2a5f32d39e7b9d WatchSource:0}: Error finding container 78d3cae2616f02d8477bdcac62923194dd3a6cf193f97cbd2d2a5f32d39e7b9d: Status 404 returned error can't find the container with id 78d3cae2616f02d8477bdcac62923194dd3a6cf193f97cbd2d2a5f32d39e7b9d
Sep 12 00:50:54.812437 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.812355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7" event={"ID":"23ff2668-da48-4a4a-a6ca-36747187b805","Type":"ContainerStarted","Data":"98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0"}
Sep 12 00:50:54.812591 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.812564    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:50:54.812649 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.812603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7" event={"ID":"23ff2668-da48-4a4a-a6ca-36747187b805","Type":"ContainerStarted","Data":"78d3cae2616f02d8477bdcac62923194dd3a6cf193f97cbd2d2a5f32d39e7b9d"}
Sep 12 00:50:54.951373 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.951337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:54.953739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:54.953713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:55.072986 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.072950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:55.345328 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.344828    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7" podStartSLOduration=2.344805131 podStartE2EDuration="2.344805131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:54.828112992 +0000 UTC m=+2057.765419512" watchObservedRunningTime="2026-09-12 00:50:55.344805131 +0000 UTC m=+2058.282111652"
Sep 12 00:50:55.346227 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.346204    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"]
Sep 12 00:50:55.348439 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:50:55.348409    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1c7e94_dd51_47c5_a8b2_4837162db0f9.slice/crio-060a9ff9befb7f3ebaba86487e834179baa96818c201fdcea360ffc006085129 WatchSource:0}: Error finding container 060a9ff9befb7f3ebaba86487e834179baa96818c201fdcea360ffc006085129: Status 404 returned error can't find the container with id 060a9ff9befb7f3ebaba86487e834179baa96818c201fdcea360ffc006085129
Sep 12 00:50:55.820047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.820015    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" event={"ID":"9c1c7e94-dd51-47c5-a8b2-4837162db0f9","Type":"ContainerStarted","Data":"32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0"}
Sep 12 00:50:55.820429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.820075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" event={"ID":"9c1c7e94-dd51-47c5-a8b2-4837162db0f9","Type":"ContainerStarted","Data":"060a9ff9befb7f3ebaba86487e834179baa96818c201fdcea360ffc006085129"}
Sep 12 00:50:55.820429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.820313    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:50:55.837743 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:50:55.837696    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" podStartSLOduration=2.447409478 podStartE2EDuration="2.837681116s" totalImagesPullingTime="390.271638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:55.836405521 +0000 UTC m=+2058.773712042" watchObservedRunningTime="2026-09-12 00:50:55.837681116 +0000 UTC m=+2058.774987636"
Sep 12 00:51:00.115322 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:00.115241    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:51:01.830784 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:01.830752    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:51:05.822549 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:05.822517    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:51:08.867168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:08.867126    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:15.503011 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:15.502974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:15.603401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:15.603367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:16.435198 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:16.435166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:17.271340 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:17.271303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:18.038423 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:18.038379    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:18.848944 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:18.848869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:19.526022 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:19.525977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:20.207667 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:20.207620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:20.960670 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:20.960635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:21.936498 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:21.936420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:22.464205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.464169    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"]
Sep 12 00:51:22.464407 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.464381    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" podUID="9c1c7e94-dd51-47c5-a8b2-4837162db0f9" containerName="maas-api" containerID="cri-o://32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0" gracePeriod=30
Sep 12 00:51:22.503953 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.503911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:22.524032 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.524003    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"]
Sep 12 00:51:22.524291 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.524264    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7" podUID="23ff2668-da48-4a4a-a6ca-36747187b805" containerName="payload-processing" containerID="cri-o://98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0" gracePeriod=30
Sep 12 00:51:22.603034 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.603004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:22.693307 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.693177    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:22.765674 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.765631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:22.799793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.799771    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:51:22.803173 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.803153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:51:22.812021 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812001    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bzwq\" (UniqueName: \"kubernetes.io/projected/23ff2668-da48-4a4a-a6ca-36747187b805-kube-api-access-7bzwq\") pod \"23ff2668-da48-4a4a-a6ca-36747187b805\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") "
Sep 12 00:51:22.812126 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812051    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-metrics-certs\") pod \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") "
Sep 12 00:51:22.812193 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812164    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") pod \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") "
Sep 12 00:51:22.812252 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pwsv\" (UniqueName: \"kubernetes.io/projected/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-kube-api-access-4pwsv\") pod \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") "
Sep 12 00:51:22.812309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-openshift-service-ca\") pod \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\" (UID: \"9c1c7e94-dd51-47c5-a8b2-4837162db0f9\") "
Sep 12 00:51:22.812309 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812284    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ff2668-da48-4a4a-a6ca-36747187b805-plugins-config-volume\") pod \"23ff2668-da48-4a4a-a6ca-36747187b805\" (UID: \"23ff2668-da48-4a4a-a6ca-36747187b805\") "
Sep 12 00:51:22.812641 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812614    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c1c7e94-dd51-47c5-a8b2-4837162db0f9" (UID: "9c1c7e94-dd51-47c5-a8b2-4837162db0f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:22.812748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.812706    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ff2668-da48-4a4a-a6ca-36747187b805-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23ff2668-da48-4a4a-a6ca-36747187b805" (UID: "23ff2668-da48-4a4a-a6ca-36747187b805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:22.814205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.814172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c1c7e94-dd51-47c5-a8b2-4837162db0f9" (UID: "9c1c7e94-dd51-47c5-a8b2-4837162db0f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:51:22.814205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.814185    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c1c7e94-dd51-47c5-a8b2-4837162db0f9" (UID: "9c1c7e94-dd51-47c5-a8b2-4837162db0f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:51:22.814353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.814249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ff2668-da48-4a4a-a6ca-36747187b805-kube-api-access-7bzwq" (OuterVolumeSpecName: "kube-api-access-7bzwq") pod "23ff2668-da48-4a4a-a6ca-36747187b805" (UID: "23ff2668-da48-4a4a-a6ca-36747187b805"). InnerVolumeSpecName "kube-api-access-7bzwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:22.814353 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.814278    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-kube-api-access-4pwsv" (OuterVolumeSpecName: "kube-api-access-4pwsv") pod "9c1c7e94-dd51-47c5-a8b2-4837162db0f9" (UID: "9c1c7e94-dd51-47c5-a8b2-4837162db0f9"). InnerVolumeSpecName "kube-api-access-4pwsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:22.913535 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.913504    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzwq\" (UniqueName: \"kubernetes.io/projected/23ff2668-da48-4a4a-a6ca-36747187b805-kube-api-access-7bzwq\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:22.913535 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.913530    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:22.913535 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.913540    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:22.913752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.913548    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwsv\" (UniqueName: \"kubernetes.io/projected/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-kube-api-access-4pwsv\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:22.913752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.913557    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c7e94-dd51-47c5-a8b2-4837162db0f9-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:22.913752 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.913566    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ff2668-da48-4a4a-a6ca-36747187b805-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:22.964575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.964546    2562 generic.go:356] "Generic (PLEG): container finished" podID="23ff2668-da48-4a4a-a6ca-36747187b805" containerID="98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0" exitCode=0
Sep 12 00:51:22.964950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.964592    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"
Sep 12 00:51:22.964950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.964632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7" event={"ID":"23ff2668-da48-4a4a-a6ca-36747187b805","Type":"ContainerDied","Data":"98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0"}
Sep 12 00:51:22.964950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.964685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7" event={"ID":"23ff2668-da48-4a4a-a6ca-36747187b805","Type":"ContainerDied","Data":"78d3cae2616f02d8477bdcac62923194dd3a6cf193f97cbd2d2a5f32d39e7b9d"}
Sep 12 00:51:22.964950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.964705    2562 scope.go:117] "RemoveContainer" containerID="98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0"
Sep 12 00:51:22.966101 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.966079    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c1c7e94-dd51-47c5-a8b2-4837162db0f9" containerID="32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0" exitCode=0
Sep 12 00:51:22.966264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.966110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" event={"ID":"9c1c7e94-dd51-47c5-a8b2-4837162db0f9","Type":"ContainerDied","Data":"32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0"}
Sep 12 00:51:22.966264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.966139    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"
Sep 12 00:51:22.966264 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.966151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s" event={"ID":"9c1c7e94-dd51-47c5-a8b2-4837162db0f9","Type":"ContainerDied","Data":"060a9ff9befb7f3ebaba86487e834179baa96818c201fdcea360ffc006085129"}
Sep 12 00:51:22.974004 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.973986    2562 scope.go:117] "RemoveContainer" containerID="98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0"
Sep 12 00:51:22.974318 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:51:22.974301    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0\": container with ID starting with 98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0 not found: ID does not exist" containerID="98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0"
Sep 12 00:51:22.974379 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.974326    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0"} err="failed to get container status \"98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0\": rpc error: code = NotFound desc = could not find container \"98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0\": container with ID starting with 98897e220e8ba1706f51998bf7084b13f9fb3fa17b177d5b206880ff64e810d0 not found: ID does not exist"
Sep 12 00:51:22.974379 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.974342    2562 scope.go:117] "RemoveContainer" containerID="32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0"
Sep 12 00:51:22.982962 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.982947    2562 scope.go:117] "RemoveContainer" containerID="32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0"
Sep 12 00:51:22.983203 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:51:22.983184    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0\": container with ID starting with 32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0 not found: ID does not exist" containerID="32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0"
Sep 12 00:51:22.983269 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.983208    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0"} err="failed to get container status \"32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0\": rpc error: code = NotFound desc = could not find container \"32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0\": container with ID starting with 32c14374d84128c01725067770dcfa9bdfe4b8d5b3dbec0c08ce50ebf47c24b0 not found: ID does not exist"
Sep 12 00:51:22.990976 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.990953    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"]
Sep 12 00:51:22.996992 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:22.996972    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b4b570-54d5bf54b9-64m8s"]
Sep 12 00:51:23.007155 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.007135    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"]
Sep 12 00:51:23.010714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.010693    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b4b570-9ccbbb667-4x2q7"]
Sep 12 00:51:23.072000 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.071969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:23.172050 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.172007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:23.571755 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.571723    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ff2668-da48-4a4a-a6ca-36747187b805" path="/var/lib/kubelet/pods/23ff2668-da48-4a4a-a6ca-36747187b805/volumes"
Sep 12 00:51:23.572123 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.572111    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c1c7e94-dd51-47c5-a8b2-4837162db0f9" path="/var/lib/kubelet/pods/9c1c7e94-dd51-47c5-a8b2-4837162db0f9/volumes"
Sep 12 00:51:23.845365 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:23.845332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:24.649013 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:24.648975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:25.411717 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:25.411686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:26.112799 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:26.112764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:26.812852 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:26.812814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:51:29.005625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:29.005584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:29.073766 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:29.073732    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:29.279337 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:29.279262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:30.933488 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:30.933437    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:33.170219 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:33.170186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:34.049989 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:34.049960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:34.867168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:34.867119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:35.503267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:35.503226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:36.352361 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:36.352290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:37.224804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:37.224775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:37.639504 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:37.639455    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:51:37.652509 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:37.652486    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:51:37.880621 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:37.880589    2562 scope.go:117] "RemoveContainer" containerID="5bbc4022449c69a09ddabf5a07ea7084c38a1749f49eb8efb30f77ebe4993552"
Sep 12 00:51:37.888893 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:37.888875    2562 scope.go:117] "RemoveContainer" containerID="43863a2b15d472871a2dd2b23336b61ce19d615004e964e1693c50c0d9f532f1"
Sep 12 00:51:46.416940 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:46.416901    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:51:48.803448 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:48.803411    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"]
Sep 12 00:51:48.803948 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:48.803645    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" podUID="efce3f42-da26-46cb-88dd-5b697f7c62ae" containerName="maas-api" containerID="cri-o://25060d8280c79d46fb77fef152566eb03db7d5082ef1479e824ef6371b6454ca" gracePeriod=30
Sep 12 00:51:48.838253 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:48.838225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:48.869196 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:48.869165    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"]
Sep 12 00:51:48.869464 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:48.869439    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx" podUID="4417e401-c8cc-4522-88a3-d2c42c1a4bfe" containerName="payload-pre-processing" containerID="cri-o://7489b31cab9c2e9128fcb45951fcbc42762c6b1c7d5eae3827280befe421defb" gracePeriod=30
Sep 12 00:51:48.938895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:48.938866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:49.041323 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.041272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:49.102780 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.102752    2562 generic.go:356] "Generic (PLEG): container finished" podID="efce3f42-da26-46cb-88dd-5b697f7c62ae" containerID="25060d8280c79d46fb77fef152566eb03db7d5082ef1479e824ef6371b6454ca" exitCode=0
Sep 12 00:51:49.102930 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.102832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" event={"ID":"efce3f42-da26-46cb-88dd-5b697f7c62ae","Type":"ContainerDied","Data":"25060d8280c79d46fb77fef152566eb03db7d5082ef1479e824ef6371b6454ca"}
Sep 12 00:51:49.102930 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.102899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42" event={"ID":"efce3f42-da26-46cb-88dd-5b697f7c62ae","Type":"ContainerDied","Data":"53f51cd5e08ac201825331fe630035bf8658ed36d824b7ec919898a5ec1a7217"}
Sep 12 00:51:49.102930 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.102928    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f51cd5e08ac201825331fe630035bf8658ed36d824b7ec919898a5ec1a7217"
Sep 12 00:51:49.104716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.104691    2562 generic.go:356] "Generic (PLEG): container finished" podID="4417e401-c8cc-4522-88a3-d2c42c1a4bfe" containerID="7489b31cab9c2e9128fcb45951fcbc42762c6b1c7d5eae3827280befe421defb" exitCode=0
Sep 12 00:51:49.104837 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.104726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx" event={"ID":"4417e401-c8cc-4522-88a3-d2c42c1a4bfe","Type":"ContainerDied","Data":"7489b31cab9c2e9128fcb45951fcbc42762c6b1c7d5eae3827280befe421defb"}
Sep 12 00:51:49.116182 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.116154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:49.123758 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.123738    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:51:49.144830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.144804    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:51:49.260744 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.260708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls\") pod \"efce3f42-da26-46cb-88dd-5b697f7c62ae\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") "
Sep 12 00:51:49.260929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.260770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efce3f42-da26-46cb-88dd-5b697f7c62ae-openshift-service-ca\") pod \"efce3f42-da26-46cb-88dd-5b697f7c62ae\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") "
Sep 12 00:51:49.260929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.260794    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-metrics-certs\") pod \"efce3f42-da26-46cb-88dd-5b697f7c62ae\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") "
Sep 12 00:51:49.260929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.260813    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n49mm\" (UniqueName: \"kubernetes.io/projected/efce3f42-da26-46cb-88dd-5b697f7c62ae-kube-api-access-n49mm\") pod \"efce3f42-da26-46cb-88dd-5b697f7c62ae\" (UID: \"efce3f42-da26-46cb-88dd-5b697f7c62ae\") "
Sep 12 00:51:49.261113 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.260967    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhh9z\" (UniqueName: \"kubernetes.io/projected/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-kube-api-access-nhh9z\") pod \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") "
Sep 12 00:51:49.261113 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.261070    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-plugins-config-volume\") pod \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\" (UID: \"4417e401-c8cc-4522-88a3-d2c42c1a4bfe\") "
Sep 12 00:51:49.261220 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.261139    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efce3f42-da26-46cb-88dd-5b697f7c62ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efce3f42-da26-46cb-88dd-5b697f7c62ae" (UID: "efce3f42-da26-46cb-88dd-5b697f7c62ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:49.261498 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.261449    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4417e401-c8cc-4522-88a3-d2c42c1a4bfe" (UID: "4417e401-c8cc-4522-88a3-d2c42c1a4bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:49.261639 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.261543    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efce3f42-da26-46cb-88dd-5b697f7c62ae-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:49.262937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.262917    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efce3f42-da26-46cb-88dd-5b697f7c62ae-kube-api-access-n49mm" (OuterVolumeSpecName: "kube-api-access-n49mm") pod "efce3f42-da26-46cb-88dd-5b697f7c62ae" (UID: "efce3f42-da26-46cb-88dd-5b697f7c62ae"). InnerVolumeSpecName "kube-api-access-n49mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:49.263041 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.262982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efce3f42-da26-46cb-88dd-5b697f7c62ae" (UID: "efce3f42-da26-46cb-88dd-5b697f7c62ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:51:49.263357 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.263341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "efce3f42-da26-46cb-88dd-5b697f7c62ae" (UID: "efce3f42-da26-46cb-88dd-5b697f7c62ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:51:49.263399 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.263361    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-kube-api-access-nhh9z" (OuterVolumeSpecName: "kube-api-access-nhh9z") pod "4417e401-c8cc-4522-88a3-d2c42c1a4bfe" (UID: "4417e401-c8cc-4522-88a3-d2c42c1a4bfe"). InnerVolumeSpecName "kube-api-access-nhh9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:49.362124 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.362048    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:49.362307 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.362291    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efce3f42-da26-46cb-88dd-5b697f7c62ae-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:49.362395 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.362382    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n49mm\" (UniqueName: \"kubernetes.io/projected/efce3f42-da26-46cb-88dd-5b697f7c62ae-kube-api-access-n49mm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:49.362492 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.362462    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhh9z\" (UniqueName: \"kubernetes.io/projected/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-kube-api-access-nhh9z\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:49.362591 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.362580    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4417e401-c8cc-4522-88a3-d2c42c1a4bfe-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:51:49.459421 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.459385    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:49.566546 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:49.566503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:50.112255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.112156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx" event={"ID":"4417e401-c8cc-4522-88a3-d2c42c1a4bfe","Type":"ContainerDied","Data":"56d0e3f4e2ae56e5d163eb39eb1368e947a735b2d3dae49fced62d495d353ad8"}
Sep 12 00:51:50.112255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.112191    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"
Sep 12 00:51:50.112255 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.112220    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"
Sep 12 00:51:50.112901 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.112269    2562 scope.go:117] "RemoveContainer" containerID="7489b31cab9c2e9128fcb45951fcbc42762c6b1c7d5eae3827280befe421defb"
Sep 12 00:51:50.130187 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.130166    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"]
Sep 12 00:51:50.134644 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.134622    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-64e462-6869c78f6c-5mh42"]
Sep 12 00:51:50.151115 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.151085    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"]
Sep 12 00:51:50.153205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.153184    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64e462-5d477bdf5d-9k6wx"]
Sep 12 00:51:50.275026 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:50.274994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:51.044138 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:51.044099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:51.570958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:51.570926    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4417e401-c8cc-4522-88a3-d2c42c1a4bfe" path="/var/lib/kubelet/pods/4417e401-c8cc-4522-88a3-d2c42c1a4bfe/volumes"
Sep 12 00:51:51.571331 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:51.571317    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efce3f42-da26-46cb-88dd-5b697f7c62ae" path="/var/lib/kubelet/pods/efce3f42-da26-46cb-88dd-5b697f7c62ae/volumes"
Sep 12 00:51:51.794581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:51.794547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:54.555817 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:54.555779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:51:59.315887 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:51:59.315848    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:52:31.125899 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:31.125856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:31.817657 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:31.817618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:37.605144 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:37.605099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:37.699729 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:37.699692    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:37.879739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:37.879654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:37.917714 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:37.917681    2562 scope.go:117] "RemoveContainer" containerID="cb2ed5304ae9f1de6443aad1746f10c9b7cb6fa618c03dcd3bedfdf86b8dd4d3"
Sep 12 00:52:38.258387 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.258290    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"]
Sep 12 00:52:38.259105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259032    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4417e401-c8cc-4522-88a3-d2c42c1a4bfe" containerName="payload-pre-processing"
Sep 12 00:52:38.259105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259064    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4417e401-c8cc-4522-88a3-d2c42c1a4bfe" containerName="payload-pre-processing"
Sep 12 00:52:38.259105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259077    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ff2668-da48-4a4a-a6ca-36747187b805" containerName="payload-processing"
Sep 12 00:52:38.259105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259086    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ff2668-da48-4a4a-a6ca-36747187b805" containerName="payload-processing"
Sep 12 00:52:38.259105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259101    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1c7e94-dd51-47c5-a8b2-4837162db0f9" containerName="maas-api"
Sep 12 00:52:38.259105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259110    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1c7e94-dd51-47c5-a8b2-4837162db0f9" containerName="maas-api"
Sep 12 00:52:38.259460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259149    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efce3f42-da26-46cb-88dd-5b697f7c62ae" containerName="maas-api"
Sep 12 00:52:38.259460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259159    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="efce3f42-da26-46cb-88dd-5b697f7c62ae" containerName="maas-api"
Sep 12 00:52:38.259460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259252    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4417e401-c8cc-4522-88a3-d2c42c1a4bfe" containerName="payload-pre-processing"
Sep 12 00:52:38.259460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259263    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="efce3f42-da26-46cb-88dd-5b697f7c62ae" containerName="maas-api"
Sep 12 00:52:38.259460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259274    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1c7e94-dd51-47c5-a8b2-4837162db0f9" containerName="maas-api"
Sep 12 00:52:38.259460 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.259283    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ff2668-da48-4a4a-a6ca-36747187b805" containerName="payload-processing"
Sep 12 00:52:38.263952 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.263927    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"]
Sep 12 00:52:38.264065 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.264013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:38.266632 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.266607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:52:38.415293 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.415260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d86aa-2961-4738-b02e-34532a228528-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94\" (UID: \"da3d86aa-2961-4738-b02e-34532a228528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:38.516497 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.516399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d86aa-2961-4738-b02e-34532a228528-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94\" (UID: \"da3d86aa-2961-4738-b02e-34532a228528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:38.517070 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.517050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d86aa-2961-4738-b02e-34532a228528-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94\" (UID: \"da3d86aa-2961-4738-b02e-34532a228528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:38.574541 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.574509    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:38.630051 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.630010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:38.847259 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:38.847233    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"]
Sep 12 00:52:38.849907 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:52:38.849880    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3d86aa_2961_4738_b02e_34532a228528.slice/crio-584a14ffb3a00fcb1e0c707f73aa64976eaef8fabb39b7a286ce81a388836bd7 WatchSource:0}: Error finding container 584a14ffb3a00fcb1e0c707f73aa64976eaef8fabb39b7a286ce81a388836bd7: Status 404 returned error can't find the container with id 584a14ffb3a00fcb1e0c707f73aa64976eaef8fabb39b7a286ce81a388836bd7
Sep 12 00:52:39.358495 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:39.358445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:39.374973 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:39.374943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94" event={"ID":"da3d86aa-2961-4738-b02e-34532a228528","Type":"ContainerStarted","Data":"673553a4690e77e32169ebb853a8b834f8a6850a1f97d11aafbad9dc60c25319"}
Sep 12 00:52:39.375134 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:39.374992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94" event={"ID":"da3d86aa-2961-4738-b02e-34532a228528","Type":"ContainerStarted","Data":"584a14ffb3a00fcb1e0c707f73aa64976eaef8fabb39b7a286ce81a388836bd7"}
Sep 12 00:52:39.391898 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:39.391852    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94" podStartSLOduration=1.3918364890000001 podStartE2EDuration="1.391836489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:39.389375572 +0000 UTC m=+2162.326682105" watchObservedRunningTime="2026-09-12 00:52:39.391836489 +0000 UTC m=+2162.329143008"
Sep 12 00:52:40.121136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:40.121089    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:40.384130 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:40.384038    2562 generic.go:356] "Generic (PLEG): container finished" podID="da3d86aa-2961-4738-b02e-34532a228528" containerID="673553a4690e77e32169ebb853a8b834f8a6850a1f97d11aafbad9dc60c25319" exitCode=0
Sep 12 00:52:40.384130 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:40.384083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94" event={"ID":"da3d86aa-2961-4738-b02e-34532a228528","Type":"ContainerDied","Data":"673553a4690e77e32169ebb853a8b834f8a6850a1f97d11aafbad9dc60c25319"}
Sep 12 00:52:41.122617 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:41.122575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:41.513658 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:41.513632    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:41.545510 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:41.545458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d86aa-2961-4738-b02e-34532a228528-maas-api-service-ca\") pod \"da3d86aa-2961-4738-b02e-34532a228528\" (UID: \"da3d86aa-2961-4738-b02e-34532a228528\") "
Sep 12 00:52:41.545771 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:41.545752    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3d86aa-2961-4738-b02e-34532a228528-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da3d86aa-2961-4738-b02e-34532a228528" (UID: "da3d86aa-2961-4738-b02e-34532a228528"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:52:41.646910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:41.646844    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d86aa-2961-4738-b02e-34532a228528-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:41.899075 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:41.899000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:42.395069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:42.395043    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"
Sep 12 00:52:42.395438 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:42.395037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94" event={"ID":"da3d86aa-2961-4738-b02e-34532a228528","Type":"ContainerDied","Data":"584a14ffb3a00fcb1e0c707f73aa64976eaef8fabb39b7a286ce81a388836bd7"}
Sep 12 00:52:42.395438 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:42.395170    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="584a14ffb3a00fcb1e0c707f73aa64976eaef8fabb39b7a286ce81a388836bd7"
Sep 12 00:52:48.717964 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.717924    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"]
Sep 12 00:52:48.718371 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.718176    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2" podUID="897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" containerName="maas-api" containerID="cri-o://101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4" gracePeriod=30
Sep 12 00:52:48.749954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.749917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:48.771850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.771808    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"]
Sep 12 00:52:48.772597 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.772121    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598" podUID="a386bdfa-89b5-4882-a387-467eacd29bda" containerName="payload-processing" containerID="cri-o://44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42" gracePeriod=30
Sep 12 00:52:48.779112 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.779078    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"]
Sep 12 00:52:48.779594 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.779563    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd" podUID="74ec62c0-ce47-4ad9-bc19-48e5a065ce57" containerName="payload-pre-processing" containerID="cri-o://a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630" gracePeriod=30
Sep 12 00:52:48.813698 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.813659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:48.875850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.875817    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:48.934705 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.934669    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"]
Sep 12 00:52:48.937515 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.937415    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbe1c-blk94"]
Sep 12 00:52:48.943441 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:48.943412    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:49.003513 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.003431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:49.052784 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.052752    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:52:49.066829 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.066796    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:52:49.075299 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.075267    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:52:49.115294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115260    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-plugins-config-volume\") pod \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") "
Sep 12 00:52:49.115294 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqb4p\" (UniqueName: \"kubernetes.io/projected/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-kube-api-access-zqb4p\") pod \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\" (UID: \"74ec62c0-ce47-4ad9-bc19-48e5a065ce57\") "
Sep 12 00:52:49.115572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115331    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-maas-api-tls\") pod \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") "
Sep 12 00:52:49.115572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115373    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-openshift-service-ca\") pod \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") "
Sep 12 00:52:49.115572 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115551    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c5n9\" (UniqueName: \"kubernetes.io/projected/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-kube-api-access-8c5n9\") pod \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") "
Sep 12 00:52:49.115757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a386bdfa-89b5-4882-a387-467eacd29bda-plugins-config-volume\") pod \"a386bdfa-89b5-4882-a387-467eacd29bda\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") "
Sep 12 00:52:49.115757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d98gn\" (UniqueName: \"kubernetes.io/projected/a386bdfa-89b5-4882-a387-467eacd29bda-kube-api-access-d98gn\") pod \"a386bdfa-89b5-4882-a387-467eacd29bda\" (UID: \"a386bdfa-89b5-4882-a387-467eacd29bda\") "
Sep 12 00:52:49.115757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115698    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-metrics-certs\") pod \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\" (UID: \"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2\") "
Sep 12 00:52:49.115757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74ec62c0-ce47-4ad9-bc19-48e5a065ce57" (UID: "74ec62c0-ce47-4ad9-bc19-48e5a065ce57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:52:49.115757 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.115737    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" (UID: "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:52:49.116152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.116103    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.116152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.116127    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.116152 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.116126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a386bdfa-89b5-4882-a387-467eacd29bda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a386bdfa-89b5-4882-a387-467eacd29bda" (UID: "a386bdfa-89b5-4882-a387-467eacd29bda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:52:49.117713 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.117677    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" (UID: "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:52:49.118023 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.117999    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-kube-api-access-zqb4p" (OuterVolumeSpecName: "kube-api-access-zqb4p") pod "74ec62c0-ce47-4ad9-bc19-48e5a065ce57" (UID: "74ec62c0-ce47-4ad9-bc19-48e5a065ce57"). InnerVolumeSpecName "kube-api-access-zqb4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:52:49.118125 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.118033    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a386bdfa-89b5-4882-a387-467eacd29bda-kube-api-access-d98gn" (OuterVolumeSpecName: "kube-api-access-d98gn") pod "a386bdfa-89b5-4882-a387-467eacd29bda" (UID: "a386bdfa-89b5-4882-a387-467eacd29bda"). InnerVolumeSpecName "kube-api-access-d98gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:52:49.118216 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.118120    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-kube-api-access-8c5n9" (OuterVolumeSpecName: "kube-api-access-8c5n9") pod "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" (UID: "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2"). InnerVolumeSpecName "kube-api-access-8c5n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:52:49.118289 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.118266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" (UID: "897c93de-6e20-4c96-adc2-fc8d2aa3e2b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:52:49.217565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.217527    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a386bdfa-89b5-4882-a387-467eacd29bda-plugins-config-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.217565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.217559    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d98gn\" (UniqueName: \"kubernetes.io/projected/a386bdfa-89b5-4882-a387-467eacd29bda-kube-api-access-d98gn\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.217565 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.217569    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.217812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.217578    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqb4p\" (UniqueName: \"kubernetes.io/projected/74ec62c0-ce47-4ad9-bc19-48e5a065ce57-kube-api-access-zqb4p\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.217812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.217587    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.217812 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.217595    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5n9\" (UniqueName: \"kubernetes.io/projected/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2-kube-api-access-8c5n9\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:49.250762 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.250669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:49.358707 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.358662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:49.432626 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.432594    2562 generic.go:356] "Generic (PLEG): container finished" podID="897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" containerID="101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4" exitCode=0
Sep 12 00:52:49.432804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.432637    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"
Sep 12 00:52:49.432804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.432666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2" event={"ID":"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2","Type":"ContainerDied","Data":"101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4"}
Sep 12 00:52:49.432804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.432734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2" event={"ID":"897c93de-6e20-4c96-adc2-fc8d2aa3e2b2","Type":"ContainerDied","Data":"2b74c6acbc2bec9488de8719028a00dbc8b934c06421a58da52cee3d9b8ae5ba"}
Sep 12 00:52:49.432804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.432759    2562 scope.go:117] "RemoveContainer" containerID="101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4"
Sep 12 00:52:49.434103 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.434081    2562 generic.go:356] "Generic (PLEG): container finished" podID="74ec62c0-ce47-4ad9-bc19-48e5a065ce57" containerID="a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630" exitCode=0
Sep 12 00:52:49.434199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.434133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"
Sep 12 00:52:49.434199 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.434166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd" event={"ID":"74ec62c0-ce47-4ad9-bc19-48e5a065ce57","Type":"ContainerDied","Data":"a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630"}
Sep 12 00:52:49.434310 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.434212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd" event={"ID":"74ec62c0-ce47-4ad9-bc19-48e5a065ce57","Type":"ContainerDied","Data":"3d199186fa55ad46f7988d97dec49f5dd804333ce85cf1d322ae57429f48af4d"}
Sep 12 00:52:49.435737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.435712    2562 generic.go:356] "Generic (PLEG): container finished" podID="a386bdfa-89b5-4882-a387-467eacd29bda" containerID="44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42" exitCode=0
Sep 12 00:52:49.435866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.435756    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"
Sep 12 00:52:49.435866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.435764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598" event={"ID":"a386bdfa-89b5-4882-a387-467eacd29bda","Type":"ContainerDied","Data":"44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42"}
Sep 12 00:52:49.435866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.435806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598" event={"ID":"a386bdfa-89b5-4882-a387-467eacd29bda","Type":"ContainerDied","Data":"e20d5ff6d59b223fd2b3f7ae072163fa02039c9a348890b9ecdf636a9c1a586f"}
Sep 12 00:52:49.442570 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.442525    2562 scope.go:117] "RemoveContainer" containerID="101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4"
Sep 12 00:52:49.442849 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:52:49.442825    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4\": container with ID starting with 101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4 not found: ID does not exist" containerID="101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4"
Sep 12 00:52:49.442936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.442862    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4"} err="failed to get container status \"101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4\": rpc error: code = NotFound desc = could not find container \"101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4\": container with ID starting with 101e703f4c64dffe7c8b978025b221dc32035aa65a4e862a7240284f79b147a4 not found: ID does not exist"
Sep 12 00:52:49.442936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.442886    2562 scope.go:117] "RemoveContainer" containerID="a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630"
Sep 12 00:52:49.451557 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.451534    2562 scope.go:117] "RemoveContainer" containerID="a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630"
Sep 12 00:52:49.451816 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:52:49.451797    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630\": container with ID starting with a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630 not found: ID does not exist" containerID="a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630"
Sep 12 00:52:49.451867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.451822    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630"} err="failed to get container status \"a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630\": rpc error: code = NotFound desc = could not find container \"a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630\": container with ID starting with a4bf775004be8da53d8efcf888477f0e4f57b2e012fe1edd51ec2e1e96eab630 not found: ID does not exist"
Sep 12 00:52:49.451867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.451837    2562 scope.go:117] "RemoveContainer" containerID="44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42"
Sep 12 00:52:49.460136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.460116    2562 scope.go:117] "RemoveContainer" containerID="44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42"
Sep 12 00:52:49.460424 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:52:49.460402    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42\": container with ID starting with 44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42 not found: ID does not exist" containerID="44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42"
Sep 12 00:52:49.460487 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.460434    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42"} err="failed to get container status \"44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42\": rpc error: code = NotFound desc = could not find container \"44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42\": container with ID starting with 44e61257bf7bdc00a380d0f955687f3f161f49bfac64adef633cf966d3427b42 not found: ID does not exist"
Sep 12 00:52:49.466615 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.466592    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"]
Sep 12 00:52:49.469723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.469703    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbe1c-bf4c474f4-sn4dd"]
Sep 12 00:52:49.480875 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.480854    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"]
Sep 12 00:52:49.484866 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.484847    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1cbe1c-67bd869448-f9rd2"]
Sep 12 00:52:49.495553 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.495529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"]
Sep 12 00:52:49.499168 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.499145    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbe1c-ff9cb8b89-6v598"]
Sep 12 00:52:49.570865 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.570832    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ec62c0-ce47-4ad9-bc19-48e5a065ce57" path="/var/lib/kubelet/pods/74ec62c0-ce47-4ad9-bc19-48e5a065ce57/volumes"
Sep 12 00:52:49.571224 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.571211    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" path="/var/lib/kubelet/pods/897c93de-6e20-4c96-adc2-fc8d2aa3e2b2/volumes"
Sep 12 00:52:49.571555 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.571542    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a386bdfa-89b5-4882-a387-467eacd29bda" path="/var/lib/kubelet/pods/a386bdfa-89b5-4882-a387-467eacd29bda/volumes"
Sep 12 00:52:49.571839 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.571827    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3d86aa-2961-4738-b02e-34532a228528" path="/var/lib/kubelet/pods/da3d86aa-2961-4738-b02e-34532a228528/volumes"
Sep 12 00:52:49.957902 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:49.957813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:50.749737 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:50.749699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:52.010581 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:52.010547    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:52:55.201839 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.201802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:55.276178 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.276142    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:55.549778 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.549681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:55.843407 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843374    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"]
Sep 12 00:52:55.843753 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843740    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ec62c0-ce47-4ad9-bc19-48e5a065ce57" containerName="payload-pre-processing"
Sep 12 00:52:55.843800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843755    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ec62c0-ce47-4ad9-bc19-48e5a065ce57" containerName="payload-pre-processing"
Sep 12 00:52:55.843800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843776    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" containerName="maas-api"
Sep 12 00:52:55.843800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843782    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" containerName="maas-api"
Sep 12 00:52:55.843800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843794    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a386bdfa-89b5-4882-a387-467eacd29bda" containerName="payload-processing"
Sep 12 00:52:55.843800 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843800    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a386bdfa-89b5-4882-a387-467eacd29bda" containerName="payload-processing"
Sep 12 00:52:55.843954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843807    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3d86aa-2961-4738-b02e-34532a228528" containerName="revoke-keys"
Sep 12 00:52:55.843954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843812    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3d86aa-2961-4738-b02e-34532a228528" containerName="revoke-keys"
Sep 12 00:52:55.843954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843873    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3d86aa-2961-4738-b02e-34532a228528" containerName="revoke-keys"
Sep 12 00:52:55.843954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843881    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a386bdfa-89b5-4882-a387-467eacd29bda" containerName="payload-processing"
Sep 12 00:52:55.843954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843887    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="897c93de-6e20-4c96-adc2-fc8d2aa3e2b2" containerName="maas-api"
Sep 12 00:52:55.843954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.843893    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ec62c0-ce47-4ad9-bc19-48e5a065ce57" containerName="payload-pre-processing"
Sep 12 00:52:55.848299 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.848271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:55.850928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.850883    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:52:55.850928 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.850892    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:52:55.854666 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.854640    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"]
Sep 12 00:52:55.879842 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.879810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f48c8d-4524-468f-8c2e-2366a8a58e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp\" (UID: \"76f48c8d-4524-468f-8c2e-2366a8a58e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:55.981265 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.981224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f48c8d-4524-468f-8c2e-2366a8a58e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp\" (UID: \"76f48c8d-4524-468f-8c2e-2366a8a58e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:55.981938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:55.981910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f48c8d-4524-468f-8c2e-2366a8a58e22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp\" (UID: \"76f48c8d-4524-468f-8c2e-2366a8a58e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:56.162284 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:56.162184    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:56.436015 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:56.435983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"]
Sep 12 00:52:56.437956 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:52:56.437929    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f48c8d_4524_468f_8c2e_2366a8a58e22.slice/crio-d7451997eddee0446c3f894460ad900bcd3a6da54c7800bc083cc3746da0eacb WatchSource:0}: Error finding container d7451997eddee0446c3f894460ad900bcd3a6da54c7800bc083cc3746da0eacb: Status 404 returned error can't find the container with id d7451997eddee0446c3f894460ad900bcd3a6da54c7800bc083cc3746da0eacb
Sep 12 00:52:56.474540 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:56.474501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp" event={"ID":"76f48c8d-4524-468f-8c2e-2366a8a58e22","Type":"ContainerStarted","Data":"d7451997eddee0446c3f894460ad900bcd3a6da54c7800bc083cc3746da0eacb"}
Sep 12 00:52:56.666082 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:56.666043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.370056 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:57.370021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:57.481575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:57.481457    2562 generic.go:356] "Generic (PLEG): container finished" podID="76f48c8d-4524-468f-8c2e-2366a8a58e22" containerID="72d99c0b0e1c27d0935701c4c9152415e562381210b9e313f9f98cf361a9cb45" exitCode=0
Sep 12 00:52:57.481575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:57.481522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp" event={"ID":"76f48c8d-4524-468f-8c2e-2366a8a58e22","Type":"ContainerDied","Data":"72d99c0b0e1c27d0935701c4c9152415e562381210b9e313f9f98cf361a9cb45"}
Sep 12 00:52:58.020095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:58.020037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:52:58.618464 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:58.618438    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:58.710399 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:58.710357    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f48c8d-4524-468f-8c2e-2366a8a58e22-maas-api-service-ca\") pod \"76f48c8d-4524-468f-8c2e-2366a8a58e22\" (UID: \"76f48c8d-4524-468f-8c2e-2366a8a58e22\") "
Sep 12 00:52:58.710754 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:58.710729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f48c8d-4524-468f-8c2e-2366a8a58e22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76f48c8d-4524-468f-8c2e-2366a8a58e22" (UID: "76f48c8d-4524-468f-8c2e-2366a8a58e22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:52:58.811108 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:58.811022    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f48c8d-4524-468f-8c2e-2366a8a58e22-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:52:59.492938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:59.492898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp" event={"ID":"76f48c8d-4524-468f-8c2e-2366a8a58e22","Type":"ContainerDied","Data":"d7451997eddee0446c3f894460ad900bcd3a6da54c7800bc083cc3746da0eacb"}
Sep 12 00:52:59.492938 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:59.492936    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"
Sep 12 00:52:59.493157 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:52:59.492992    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7451997eddee0446c3f894460ad900bcd3a6da54c7800bc083cc3746da0eacb"
Sep 12 00:53:06.461722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.461686    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:06.533180 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.533129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:06.603515 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.603464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:06.664040 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.664005    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"]
Sep 12 00:53:06.667154 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.667128    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e88680-7cmcp"]
Sep 12 00:53:06.668979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.668954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:06.737339 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:06.737256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:07.057638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:07.057551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:07.158403 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:07.158371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:07.571221 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:07.571178    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f48c8d-4524-468f-8c2e-2366a8a58e22" path="/var/lib/kubelet/pods/76f48c8d-4524-468f-8c2e-2366a8a58e22/volumes"
Sep 12 00:53:07.847103 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:07.847019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:08.217532 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:08.217446    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:53:13.112746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.112707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:13.211196 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.211163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:13.765006 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.764973    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"]
Sep 12 00:53:13.765343 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.765331    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f48c8d-4524-468f-8c2e-2366a8a58e22" containerName="revoke-keys"
Sep 12 00:53:13.765397 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.765346    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f48c8d-4524-468f-8c2e-2366a8a58e22" containerName="revoke-keys"
Sep 12 00:53:13.765432 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.765413    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f48c8d-4524-468f-8c2e-2366a8a58e22" containerName="revoke-keys"
Sep 12 00:53:13.768184 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.768167    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:13.771857 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.771823    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rlnvx\""
Sep 12 00:53:13.771857 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.771823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:53:13.779058 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.779033    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"]
Sep 12 00:53:13.848646 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.848612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f696ff8-9b78-4333-9624-9770ad865188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj\" (UID: \"5f696ff8-9b78-4333-9624-9770ad865188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:13.949208 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.949168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f696ff8-9b78-4333-9624-9770ad865188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj\" (UID: \"5f696ff8-9b78-4333-9624-9770ad865188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:13.949817 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:13.949796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f696ff8-9b78-4333-9624-9770ad865188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj\" (UID: \"5f696ff8-9b78-4333-9624-9770ad865188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:14.078932 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:14.078902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:14.348958 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:14.348933    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"]
Sep 12 00:53:14.351113 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:53:14.351089    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f696ff8_9b78_4333_9624_9770ad865188.slice/crio-4437be534fc8c2e80baed94c29334bb90c47c6b49adf927f515c8af3d1d7f0fc WatchSource:0}: Error finding container 4437be534fc8c2e80baed94c29334bb90c47c6b49adf927f515c8af3d1d7f0fc: Status 404 returned error can't find the container with id 4437be534fc8c2e80baed94c29334bb90c47c6b49adf927f515c8af3d1d7f0fc
Sep 12 00:53:14.449482 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:14.449430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:14.569252 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:14.569224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj" event={"ID":"5f696ff8-9b78-4333-9624-9770ad865188","Type":"ContainerStarted","Data":"4437be534fc8c2e80baed94c29334bb90c47c6b49adf927f515c8af3d1d7f0fc"}
Sep 12 00:53:14.842305 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:14.842262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:15.567029 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:15.566978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:15.578462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:15.578434    2562 generic.go:356] "Generic (PLEG): container finished" podID="5f696ff8-9b78-4333-9624-9770ad865188" containerID="35234abdda838150961cd8e497b290de9bfbb4b2ae7c6fcb9c74b9dded720a64" exitCode=0
Sep 12 00:53:15.578650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:15.578528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj" event={"ID":"5f696ff8-9b78-4333-9624-9770ad865188","Type":"ContainerDied","Data":"35234abdda838150961cd8e497b290de9bfbb4b2ae7c6fcb9c74b9dded720a64"}
Sep 12 00:53:16.710580 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:16.710556    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:16.874920 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:16.874833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f696ff8-9b78-4333-9624-9770ad865188-maas-api-service-ca\") pod \"5f696ff8-9b78-4333-9624-9770ad865188\" (UID: \"5f696ff8-9b78-4333-9624-9770ad865188\") "
Sep 12 00:53:16.875233 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:16.875203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f696ff8-9b78-4333-9624-9770ad865188-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f696ff8-9b78-4333-9624-9770ad865188" (UID: "5f696ff8-9b78-4333-9624-9770ad865188"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:53:16.976296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:16.976261    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f696ff8-9b78-4333-9624-9770ad865188-maas-api-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:53:17.589265 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:17.589241    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"
Sep 12 00:53:17.589409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:17.589262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj" event={"ID":"5f696ff8-9b78-4333-9624-9770ad865188","Type":"ContainerDied","Data":"4437be534fc8c2e80baed94c29334bb90c47c6b49adf927f515c8af3d1d7f0fc"}
Sep 12 00:53:17.589409 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:17.589304    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4437be534fc8c2e80baed94c29334bb90c47c6b49adf927f515c8af3d1d7f0fc"
Sep 12 00:53:24.190804 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.190745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:24.255037 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.254998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:24.325210 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.325174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:24.379951 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.379918    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"]
Sep 12 00:53:24.382099 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.382070    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e37bfb-br8gj"]
Sep 12 00:53:24.388420 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.388395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:24.766252 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.766220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:24.875231 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:24.875197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:25.570798 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:25.570765    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f696ff8-9b78-4333-9624-9770ad865188" path="/var/lib/kubelet/pods/5f696ff8-9b78-4333-9624-9770ad865188/volumes"
Sep 12 00:53:26.707726 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:26.707694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:53:30.866130 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:30.866094    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:30.966247 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:30.966210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:31.251515 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:31.251412    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:53:35.510547 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:53:35.510506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:54:23.593308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:54:23.593270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:23.649326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:54:23.649257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:56.108222 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:54:56.108187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:54:56.949462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:54:56.949425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:54:58.447427 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:54:58.447393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:04.076544 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:04.076483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:15.964902 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:15.964863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:16.022457 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:16.022429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:55:22.609716 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:22.609680    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:55:30.506247 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:30.506166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:55:37.609699 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:37.609664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:55:37.969836 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:37.969753    2562 scope.go:117] "RemoveContainer" containerID="25060d8280c79d46fb77fef152566eb03db7d5082ef1479e824ef6371b6454ca"
Sep 12 00:55:45.012272 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:55:45.012232    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:56:01.203723 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:01.203684    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:56:06.980250 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:06.980213    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"]
Sep 12 00:56:06.980769 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:06.980422    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6" podUID="80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" containerName="manager" containerID="cri-o://b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f" gracePeriod=10
Sep 12 00:56:08.616910 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.616886    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:56:08.694765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.694736    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngwj2\" (UniqueName: \"kubernetes.io/projected/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-kube-api-access-ngwj2\") pod \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") "
Sep 12 00:56:08.694929 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.694832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-extensions-socket-volume\") pod \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\" (UID: \"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8\") "
Sep 12 00:56:08.695205 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.695174    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" (UID: "80786198-cb1b-45b6-afc2-ccf3b7b5f3a8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:56:08.696904 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.696847    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-kube-api-access-ngwj2" (OuterVolumeSpecName: "kube-api-access-ngwj2") pod "80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" (UID: "80786198-cb1b-45b6-afc2-ccf3b7b5f3a8"). InnerVolumeSpecName "kube-api-access-ngwj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:56:08.795528 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.795464    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwj2\" (UniqueName: \"kubernetes.io/projected/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-kube-api-access-ngwj2\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:56:08.795528 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:08.795522    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8-extensions-socket-volume\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:56:09.443136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.443105    2562 generic.go:356] "Generic (PLEG): container finished" podID="80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" containerID="b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f" exitCode=0
Sep 12 00:56:09.443311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.443145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6" event={"ID":"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8","Type":"ContainerDied","Data":"b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f"}
Sep 12 00:56:09.443311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.443150    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"
Sep 12 00:56:09.443311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.443183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6" event={"ID":"80786198-cb1b-45b6-afc2-ccf3b7b5f3a8","Type":"ContainerDied","Data":"9a29a9b2fc84cf3b66053a5eaca9f5e8c47d4b897881a09762057b95b9329ff3"}
Sep 12 00:56:09.443311 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.443205    2562 scope.go:117] "RemoveContainer" containerID="b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f"
Sep 12 00:56:09.452444 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.452427    2562 scope.go:117] "RemoveContainer" containerID="b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f"
Sep 12 00:56:09.452741 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:56:09.452722    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f\": container with ID starting with b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f not found: ID does not exist" containerID="b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f"
Sep 12 00:56:09.452793 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.452749    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f"} err="failed to get container status \"b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f\": rpc error: code = NotFound desc = could not find container \"b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f\": container with ID starting with b3d9f4fe4c10e9f5703407994e5db491e80566d86d52ee562dcb90de641fc63f not found: ID does not exist"
Sep 12 00:56:09.464640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.464619    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"]
Sep 12 00:56:09.468270 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.468250    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t8ch6"]
Sep 12 00:56:09.570526 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:09.570500    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" path="/var/lib/kubelet/pods/80786198-cb1b-45b6-afc2-ccf3b7b5f3a8/volumes"
Sep 12 00:56:15.301869 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:15.301835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:15.362036 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:15.361987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:31.208718 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:31.208656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:31.309693 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:31.309658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:34.432056 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:34.432020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:56:37.670273 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:37.670243    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:56:37.685450 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:56:37.685428    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:57:07.701640 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.701555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"]
Sep 12 00:57:07.702240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.701944    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" containerName="manager"
Sep 12 00:57:07.702240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.701960    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" containerName="manager"
Sep 12 00:57:07.702240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.701997    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f696ff8-9b78-4333-9624-9770ad865188" containerName="revoke-keys"
Sep 12 00:57:07.702240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.702006    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f696ff8-9b78-4333-9624-9770ad865188" containerName="revoke-keys"
Sep 12 00:57:07.702240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.702087    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="80786198-cb1b-45b6-afc2-ccf3b7b5f3a8" containerName="manager"
Sep 12 00:57:07.702240 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.702100    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f696ff8-9b78-4333-9624-9770ad865188" containerName="revoke-keys"
Sep 12 00:57:07.704890 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.704871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:07.709308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.709287    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d8v2s\""
Sep 12 00:57:07.721856 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.721827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"]
Sep 12 00:57:07.804207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.804179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47jl\" (UniqueName: \"kubernetes.io/projected/e8f5fadf-fd3d-4129-af60-2470675885b4-kube-api-access-n47jl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gspqm\" (UID: \"e8f5fadf-fd3d-4129-af60-2470675885b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:07.804361 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.804267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8f5fadf-fd3d-4129-af60-2470675885b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gspqm\" (UID: \"e8f5fadf-fd3d-4129-af60-2470675885b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:07.904880 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.904852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8f5fadf-fd3d-4129-af60-2470675885b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gspqm\" (UID: \"e8f5fadf-fd3d-4129-af60-2470675885b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:07.905019 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.904918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47jl\" (UniqueName: \"kubernetes.io/projected/e8f5fadf-fd3d-4129-af60-2470675885b4-kube-api-access-n47jl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gspqm\" (UID: \"e8f5fadf-fd3d-4129-af60-2470675885b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:07.905242 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.905220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8f5fadf-fd3d-4129-af60-2470675885b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gspqm\" (UID: \"e8f5fadf-fd3d-4129-af60-2470675885b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:07.914244 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:07.914213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47jl\" (UniqueName: \"kubernetes.io/projected/e8f5fadf-fd3d-4129-af60-2470675885b4-kube-api-access-n47jl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gspqm\" (UID: \"e8f5fadf-fd3d-4129-af60-2470675885b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:08.015649 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:08.015570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:08.297911 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:08.297881    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"]
Sep 12 00:57:08.300366 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:57:08.300342    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f5fadf_fd3d_4129_af60_2470675885b4.slice/crio-1e4bd213e48688ac1f7144415e6ef33e64ca7b3fb0433a922c155a689786f980 WatchSource:0}: Error finding container 1e4bd213e48688ac1f7144415e6ef33e64ca7b3fb0433a922c155a689786f980: Status 404 returned error can't find the container with id 1e4bd213e48688ac1f7144415e6ef33e64ca7b3fb0433a922c155a689786f980
Sep 12 00:57:08.725489 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:08.725446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm" event={"ID":"e8f5fadf-fd3d-4129-af60-2470675885b4","Type":"ContainerStarted","Data":"363fc967b687b27c420027fdf089e031aa1d88957cd5272c54fe132c75d28f76"}
Sep 12 00:57:08.725850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:08.725510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm" event={"ID":"e8f5fadf-fd3d-4129-af60-2470675885b4","Type":"ContainerStarted","Data":"1e4bd213e48688ac1f7144415e6ef33e64ca7b3fb0433a922c155a689786f980"}
Sep 12 00:57:08.725850 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:08.725551    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:08.745774 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:08.745728    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm" podStartSLOduration=1.745714108 podStartE2EDuration="1.745714108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:08.743496868 +0000 UTC m=+2431.680803379" watchObservedRunningTime="2026-09-12 00:57:08.745714108 +0000 UTC m=+2431.683020621"
Sep 12 00:57:19.731975 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:19.731933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gspqm"
Sep 12 00:57:28.112650 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:28.112616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:57:34.806207 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:34.806171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:57:43.009984 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:43.009948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:57:44.233142 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.233104    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"]
Sep 12 00:57:44.236605 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.236583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.240903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.240869    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8b139d27-kserve-self-signed-certs\""
Sep 12 00:57:44.247548 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.247517    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"]
Sep 12 00:57:44.253554 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.253523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:44.320446 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.320414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-home\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.320635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.320458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tmp-dir\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.320635 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.320577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dg2\" (UniqueName: \"kubernetes.io/projected/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-kube-api-access-d7dg2\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.320711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.320642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-dshm\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.320711 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.320661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tls-certs\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.320776 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.320712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-model-cache\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422078 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tmp-dir\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dg2\" (UniqueName: \"kubernetes.io/projected/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-kube-api-access-d7dg2\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-dshm\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tls-certs\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-model-cache\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422267 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-home\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422583 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tmp-dir\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422641 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-home\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.422641 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.422627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-model-cache\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.424574 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.424551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-dshm\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.424966 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.424942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tls-certs\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.430642 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.430623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dg2\" (UniqueName: \"kubernetes.io/projected/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-kube-api-access-d7dg2\") pod \"e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") " pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.551677 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.551573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:44.823538 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.823513    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"]
Sep 12 00:57:44.825210 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:57:44.825184    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3291fd88_0f15_4c3e_9f3d_1c9d965dc242.slice/crio-f558f7cef2b72751ded61bc5e598ffe52e7c731d3bb9a690cd6df5db8314d04c WatchSource:0}: Error finding container f558f7cef2b72751ded61bc5e598ffe52e7c731d3bb9a690cd6df5db8314d04c: Status 404 returned error can't find the container with id f558f7cef2b72751ded61bc5e598ffe52e7c731d3bb9a690cd6df5db8314d04c
Sep 12 00:57:44.908686 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.908650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk" event={"ID":"3291fd88-0f15-4c3e-9f3d-1c9d965dc242","Type":"ContainerStarted","Data":"315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235"}
Sep 12 00:57:44.908843 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.908703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk" event={"ID":"3291fd88-0f15-4c3e-9f3d-1c9d965dc242","Type":"ContainerStarted","Data":"f558f7cef2b72751ded61bc5e598ffe52e7c731d3bb9a690cd6df5db8314d04c"}
Sep 12 00:57:44.925391 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:44.925339    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk" podStartSLOduration=0.925321958 podStartE2EDuration="925.321958ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:44.923925349 +0000 UTC m=+2467.861231870" watchObservedRunningTime="2026-09-12 00:57:44.925321958 +0000 UTC m=+2467.862628478"
Sep 12 00:57:49.174170 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:49.174131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:57:51.404709 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:51.404674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:57:54.552557 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:54.552517    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:54.553049 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:54.552581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:54.564937 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:54.564910    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:57:54.975425 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:57:54.975390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:58:04.540816 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:04.540777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:14.050634 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:14.050593    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"]
Sep 12 00:58:14.117062 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:14.117019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:15.368748 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:15.368713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:58:16.099979 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:16.099942    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:58:25.118925 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:25.118892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:58:29.071107 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.071036    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk" podUID="3291fd88-0f15-4c3e-9f3d-1c9d965dc242" containerName="main" containerID="cri-o://315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235" gracePeriod=45
Sep 12 00:58:29.315954 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.315929    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:58:29.409862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.409779    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-dshm\") pod \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") "
Sep 12 00:58:29.409862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.409825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-home\") pod \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") "
Sep 12 00:58:29.409862 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.409846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7dg2\" (UniqueName: \"kubernetes.io/projected/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-kube-api-access-d7dg2\") pod \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") "
Sep 12 00:58:29.410163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.409877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-model-cache\") pod \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") "
Sep 12 00:58:29.410163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.409907    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tls-certs\") pod \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") "
Sep 12 00:58:29.410163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.409927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tmp-dir\") pod \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\" (UID: \"3291fd88-0f15-4c3e-9f3d-1c9d965dc242\") "
Sep 12 00:58:29.410163 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.410110    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-home" (OuterVolumeSpecName: "home") pod "3291fd88-0f15-4c3e-9f3d-1c9d965dc242" (UID: "3291fd88-0f15-4c3e-9f3d-1c9d965dc242"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:58:29.410354 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.410161    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-model-cache" (OuterVolumeSpecName: "model-cache") pod "3291fd88-0f15-4c3e-9f3d-1c9d965dc242" (UID: "3291fd88-0f15-4c3e-9f3d-1c9d965dc242"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:58:29.410354 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.410289    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3291fd88-0f15-4c3e-9f3d-1c9d965dc242" (UID: "3291fd88-0f15-4c3e-9f3d-1c9d965dc242"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:58:29.412166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.412143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-dshm" (OuterVolumeSpecName: "dshm") pod "3291fd88-0f15-4c3e-9f3d-1c9d965dc242" (UID: "3291fd88-0f15-4c3e-9f3d-1c9d965dc242"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:58:29.412166 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.412155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3291fd88-0f15-4c3e-9f3d-1c9d965dc242" (UID: "3291fd88-0f15-4c3e-9f3d-1c9d965dc242"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:29.412352 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.412156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-kube-api-access-d7dg2" (OuterVolumeSpecName: "kube-api-access-d7dg2") pod "3291fd88-0f15-4c3e-9f3d-1c9d965dc242" (UID: "3291fd88-0f15-4c3e-9f3d-1c9d965dc242"). InnerVolumeSpecName "kube-api-access-d7dg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:29.510590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.510554    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-dshm\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:29.510590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.510582    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-home\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:29.510590 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.510591    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dg2\" (UniqueName: \"kubernetes.io/projected/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-kube-api-access-d7dg2\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:29.510822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.510602    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-model-cache\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:29.510822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.510612    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tls-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:29.510822 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:29.510620    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3291fd88-0f15-4c3e-9f3d-1c9d965dc242-tmp-dir\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:30.144136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.144053    2562 generic.go:356] "Generic (PLEG): container finished" podID="3291fd88-0f15-4c3e-9f3d-1c9d965dc242" containerID="315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235" exitCode=0
Sep 12 00:58:30.144136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.144079    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"
Sep 12 00:58:30.144136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.144122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk" event={"ID":"3291fd88-0f15-4c3e-9f3d-1c9d965dc242","Type":"ContainerDied","Data":"315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235"}
Sep 12 00:58:30.144638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.144174    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk" event={"ID":"3291fd88-0f15-4c3e-9f3d-1c9d965dc242","Type":"ContainerDied","Data":"f558f7cef2b72751ded61bc5e598ffe52e7c731d3bb9a690cd6df5db8314d04c"}
Sep 12 00:58:30.144638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.144201    2562 scope.go:117] "RemoveContainer" containerID="315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235"
Sep 12 00:58:30.153753 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.153737    2562 scope.go:117] "RemoveContainer" containerID="315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235"
Sep 12 00:58:30.154025 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:58:30.154008    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235\": container with ID starting with 315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235 not found: ID does not exist" containerID="315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235"
Sep 12 00:58:30.154105 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.154032    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235"} err="failed to get container status \"315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235\": rpc error: code = NotFound desc = could not find container \"315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235\": container with ID starting with 315249c25ab0a63d8f9baf70a7b171a39d777ff7d64921e78b00ac672553c235 not found: ID does not exist"
Sep 12 00:58:30.162552 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.162529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"]
Sep 12 00:58:30.166575 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:30.166553    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8b139d27-kserve-bf79b658c-25fdk"]
Sep 12 00:58:31.569931 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:31.569898    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3291fd88-0f15-4c3e-9f3d-1c9d965dc242" path="/var/lib/kubelet/pods/3291fd88-0f15-4c3e-9f3d-1c9d965dc242/volumes"
Sep 12 00:58:34.111797 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.111757    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:58:34.445318 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.445235    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"]
Sep 12 00:58:34.445830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.445800    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3291fd88-0f15-4c3e-9f3d-1c9d965dc242" containerName="main"
Sep 12 00:58:34.445830 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.445825    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3291fd88-0f15-4c3e-9f3d-1c9d965dc242" containerName="main"
Sep 12 00:58:34.445997 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.445942    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3291fd88-0f15-4c3e-9f3d-1c9d965dc242" containerName="main"
Sep 12 00:58:34.450689 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.450667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.454400 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.454374    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f4scw\""
Sep 12 00:58:34.454400 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.454393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 12 00:58:34.454638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.454374    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 00:58:34.454638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.454495    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 00:58:34.454638 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.454567    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:58:34.457840 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.457803    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"]
Sep 12 00:58:34.544863 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.544828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-metrics-certs\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.545047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.544904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758bc417-9779-4dc6-920f-85f3aa42c7e8-openshift-service-ca\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.545047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.544922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwr9\" (UniqueName: \"kubernetes.io/projected/758bc417-9779-4dc6-920f-85f3aa42c7e8-kube-api-access-zxwr9\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.545047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.544947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-maas-api-tls\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.646214 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.646181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-metrics-certs\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.646403 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.646257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758bc417-9779-4dc6-920f-85f3aa42c7e8-openshift-service-ca\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.646403 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.646281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwr9\" (UniqueName: \"kubernetes.io/projected/758bc417-9779-4dc6-920f-85f3aa42c7e8-kube-api-access-zxwr9\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.646403 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.646318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-maas-api-tls\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.647140 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.647109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758bc417-9779-4dc6-920f-85f3aa42c7e8-openshift-service-ca\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.648913 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.648882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-metrics-certs\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.649013 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.648892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-maas-api-tls\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.654936 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.654912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwr9\" (UniqueName: \"kubernetes.io/projected/758bc417-9779-4dc6-920f-85f3aa42c7e8-kube-api-access-zxwr9\") pod \"maas-api-6b768f88d8-cfs7q\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") " pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:34.763656 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:34.763569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:35.039297 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:35.039270    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"]
Sep 12 00:58:35.041331 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:58:35.041306    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758bc417_9779_4dc6_920f_85f3aa42c7e8.slice/crio-f3d6c8214a9c8ac0b7d3ad354166edf650d3c949978bd28ae2fabc0c3feecb7e WatchSource:0}: Error finding container f3d6c8214a9c8ac0b7d3ad354166edf650d3c949978bd28ae2fabc0c3feecb7e: Status 404 returned error can't find the container with id f3d6c8214a9c8ac0b7d3ad354166edf650d3c949978bd28ae2fabc0c3feecb7e
Sep 12 00:58:35.043109 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:35.043087    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:58:35.173296 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:35.173257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q" event={"ID":"758bc417-9779-4dc6-920f-85f3aa42c7e8","Type":"ContainerStarted","Data":"f3d6c8214a9c8ac0b7d3ad354166edf650d3c949978bd28ae2fabc0c3feecb7e"}
Sep 12 00:58:36.179671 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:36.179631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q" event={"ID":"758bc417-9779-4dc6-920f-85f3aa42c7e8","Type":"ContainerStarted","Data":"663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f"}
Sep 12 00:58:36.180066 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:36.179743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:36.200718 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:36.200670    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q" podStartSLOduration=1.7758455579999999 podStartE2EDuration="2.200654761s" totalImagesPullingTime="424.809203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:36.199286657 +0000 UTC m=+2519.136593176" watchObservedRunningTime="2026-09-12 00:58:36.200654761 +0000 UTC m=+2519.137961280"
Sep 12 00:58:41.836991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:41.836956    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"]
Sep 12 00:58:41.837429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:41.837232    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q" podUID="758bc417-9779-4dc6-920f-85f3aa42c7e8" containerName="maas-api" containerID="cri-o://663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f" gracePeriod=30
Sep 12 00:58:41.842725 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:41.842702    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:42.085529 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.085503    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:42.109117 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.109000    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxwr9\" (UniqueName: \"kubernetes.io/projected/758bc417-9779-4dc6-920f-85f3aa42c7e8-kube-api-access-zxwr9\") pod \"758bc417-9779-4dc6-920f-85f3aa42c7e8\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") "
Sep 12 00:58:42.109117 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.109069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-maas-api-tls\") pod \"758bc417-9779-4dc6-920f-85f3aa42c7e8\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") "
Sep 12 00:58:42.109867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.109117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758bc417-9779-4dc6-920f-85f3aa42c7e8-openshift-service-ca\") pod \"758bc417-9779-4dc6-920f-85f3aa42c7e8\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") "
Sep 12 00:58:42.109867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.109219    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-metrics-certs\") pod \"758bc417-9779-4dc6-920f-85f3aa42c7e8\" (UID: \"758bc417-9779-4dc6-920f-85f3aa42c7e8\") "
Sep 12 00:58:42.109867 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.109641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758bc417-9779-4dc6-920f-85f3aa42c7e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "758bc417-9779-4dc6-920f-85f3aa42c7e8" (UID: "758bc417-9779-4dc6-920f-85f3aa42c7e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:42.111339 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.111311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "758bc417-9779-4dc6-920f-85f3aa42c7e8" (UID: "758bc417-9779-4dc6-920f-85f3aa42c7e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:42.111339 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.111327    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758bc417-9779-4dc6-920f-85f3aa42c7e8-kube-api-access-zxwr9" (OuterVolumeSpecName: "kube-api-access-zxwr9") pod "758bc417-9779-4dc6-920f-85f3aa42c7e8" (UID: "758bc417-9779-4dc6-920f-85f3aa42c7e8"). InnerVolumeSpecName "kube-api-access-zxwr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:42.111558 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.111396    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "758bc417-9779-4dc6-920f-85f3aa42c7e8" (UID: "758bc417-9779-4dc6-920f-85f3aa42c7e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:58:42.209722 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209686    2562 generic.go:356] "Generic (PLEG): container finished" podID="758bc417-9779-4dc6-920f-85f3aa42c7e8" containerID="663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f" exitCode=0
Sep 12 00:58:42.209884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209744    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"
Sep 12 00:58:42.209884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209781    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q" event={"ID":"758bc417-9779-4dc6-920f-85f3aa42c7e8","Type":"ContainerDied","Data":"663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f"}
Sep 12 00:58:42.209884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q" event={"ID":"758bc417-9779-4dc6-920f-85f3aa42c7e8","Type":"ContainerDied","Data":"f3d6c8214a9c8ac0b7d3ad354166edf650d3c949978bd28ae2fabc0c3feecb7e"}
Sep 12 00:58:42.209884 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209864    2562 scope.go:117] "RemoveContainer" containerID="663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f"
Sep 12 00:58:42.210068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209979    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-metrics-certs\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:42.210068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.209998    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwr9\" (UniqueName: \"kubernetes.io/projected/758bc417-9779-4dc6-920f-85f3aa42c7e8-kube-api-access-zxwr9\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:42.210068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.210012    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758bc417-9779-4dc6-920f-85f3aa42c7e8-maas-api-tls\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:42.210068 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.210025    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758bc417-9779-4dc6-920f-85f3aa42c7e8-openshift-service-ca\") on node \"ip-10-0-140-187.ec2.internal\" DevicePath \"\""
Sep 12 00:58:42.218103 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.218075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:58:42.220275 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.220257    2562 scope.go:117] "RemoveContainer" containerID="663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f"
Sep 12 00:58:42.220609 ip-10-0-140-187 kubenswrapper[2562]: E0912 00:58:42.220588    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f\": container with ID starting with 663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f not found: ID does not exist" containerID="663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f"
Sep 12 00:58:42.220666 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.220619    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f"} err="failed to get container status \"663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f\": rpc error: code = NotFound desc = could not find container \"663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f\": container with ID starting with 663aebf18da369ace3101831203c4586565a18742f76de0f24cc35709768e20f not found: ID does not exist"
Sep 12 00:58:42.238783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.238755    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"]
Sep 12 00:58:42.244100 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:42.244078    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b768f88d8-cfs7q"]
Sep 12 00:58:43.570779 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:43.570739    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="758bc417-9779-4dc6-920f-85f3aa42c7e8" path="/var/lib/kubelet/pods/758bc417-9779-4dc6-920f-85f3aa42c7e8/volumes"
Sep 12 00:58:49.104765 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:49.104728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:58:55.010097 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:58:55.010054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:59:00.867713 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:00.867677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:01.700084 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:01.700051    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5k2j"]
Sep 12 00:59:10.860069 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:10.860040    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446f459d6-p9cwf_c4e6a6c5-8374-4047-8472-7372b0aed510/copy-manifests/0.log"
Sep 12 00:59:10.872260 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:10.872232    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446f459d6-p9cwf_c4e6a6c5-8374-4047-8472-7372b0aed510/manager/0.log"
Sep 12 00:59:12.758914 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:12.758884    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-h7ptz_8a79b70d-f67d-4dc1-b8e1-89c66440c9f7/manager/0.log"
Sep 12 00:59:12.887095 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:12.887068    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tf8t9_56571bde-c5fa-4aae-a2b2-67a1a5234cd5/kuadrant-console-plugin/0.log"
Sep 12 00:59:13.013089 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:13.013010    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gspqm_e8f5fadf-fd3d-4129-af60-2470675885b4/manager/0.log"
Sep 12 00:59:13.137903 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:13.137870    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t5k2j_ce17ca8a-f51c-4ca3-a856-9398789d4dca/limitador/0.log"
Sep 12 00:59:13.263986 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:13.263914    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-n6q84_fd3180a5-2d82-4349-862b-ce5789e11ca1/manager/0.log"
Sep 12 00:59:14.835764 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:14.835731    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jmrcv_e6e4afa3-ef98-450c-bae3-e9da230afc8c/main/0.log"
Sep 12 00:59:15.203326 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:15.203250    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rj8ts_aaa3efd0-a92a-4fc4-a28f-8526d4181c06/main/0.log"
Sep 12 00:59:15.560411 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:15.560334    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2z7xn_fd8e3e9e-09cb-44b0-80fa-56aa190b2cad/main/0.log"
Sep 12 00:59:20.774521 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.774483    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wvbw5/must-gather-g64rj"]
Sep 12 00:59:20.774991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.774862    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758bc417-9779-4dc6-920f-85f3aa42c7e8" containerName="maas-api"
Sep 12 00:59:20.774991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.774874    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="758bc417-9779-4dc6-920f-85f3aa42c7e8" containerName="maas-api"
Sep 12 00:59:20.774991 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.774944    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="758bc417-9779-4dc6-920f-85f3aa42c7e8" containerName="maas-api"
Sep 12 00:59:20.778038 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.778020    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:20.780490 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.780451    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvbw5\"/\"kube-root-ca.crt\""
Sep 12 00:59:20.780588 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.780524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wvbw5\"/\"default-dockercfg-wwsd6\""
Sep 12 00:59:20.781502 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.781484    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvbw5\"/\"openshift-service-ca.crt\""
Sep 12 00:59:20.786614 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.786592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvbw5/must-gather-g64rj"]
Sep 12 00:59:20.792719 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.792698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:20.886011 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.885971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log" limit=1024
Sep 12 00:59:20.954636 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.954606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcldt\" (UniqueName: \"kubernetes.io/projected/69b85226-16b0-4661-8bc6-7d0a0236630a-kube-api-access-lcldt\") pod \"must-gather-g64rj\" (UID: \"69b85226-16b0-4661-8bc6-7d0a0236630a\") " pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:20.954829 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:20.954639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69b85226-16b0-4661-8bc6-7d0a0236630a-must-gather-output\") pod \"must-gather-g64rj\" (UID: \"69b85226-16b0-4661-8bc6-7d0a0236630a\") " pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:21.055136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.055054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcldt\" (UniqueName: \"kubernetes.io/projected/69b85226-16b0-4661-8bc6-7d0a0236630a-kube-api-access-lcldt\") pod \"must-gather-g64rj\" (UID: \"69b85226-16b0-4661-8bc6-7d0a0236630a\") " pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:21.055136 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.055095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69b85226-16b0-4661-8bc6-7d0a0236630a-must-gather-output\") pod \"must-gather-g64rj\" (UID: \"69b85226-16b0-4661-8bc6-7d0a0236630a\") " pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:21.055439 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.055418    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69b85226-16b0-4661-8bc6-7d0a0236630a-must-gather-output\") pod \"must-gather-g64rj\" (UID: \"69b85226-16b0-4661-8bc6-7d0a0236630a\") " pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:21.063449 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.063419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcldt\" (UniqueName: \"kubernetes.io/projected/69b85226-16b0-4661-8bc6-7d0a0236630a-kube-api-access-lcldt\") pod \"must-gather-g64rj\" (UID: \"69b85226-16b0-4661-8bc6-7d0a0236630a\") " pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:21.088086 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.088064    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvbw5/must-gather-g64rj"
Sep 12 00:59:21.367429 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.367405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvbw5/must-gather-g64rj"]
Sep 12 00:59:21.369242 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:59:21.369211    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69b85226_16b0_4661_8bc6_7d0a0236630a.slice/crio-f778689f3f103f691bed5d22c2a0304418a4e90a4d84e04fc5bf7bfee9746e9f WatchSource:0}: Error finding container f778689f3f103f691bed5d22c2a0304418a4e90a4d84e04fc5bf7bfee9746e9f: Status 404 returned error can't find the container with id f778689f3f103f691bed5d22c2a0304418a4e90a4d84e04fc5bf7bfee9746e9f
Sep 12 00:59:21.399413 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:21.399387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvbw5/must-gather-g64rj" event={"ID":"69b85226-16b0-4661-8bc6-7d0a0236630a","Type":"ContainerStarted","Data":"f778689f3f103f691bed5d22c2a0304418a4e90a4d84e04fc5bf7bfee9746e9f"}
Sep 12 00:59:22.406760 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:22.406727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvbw5/must-gather-g64rj" event={"ID":"69b85226-16b0-4661-8bc6-7d0a0236630a","Type":"ContainerStarted","Data":"f3aed1e0676d7b38b1624792013eac5c6e613608355147237dc87be78652d12f"}
Sep 12 00:59:22.407119 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:22.406771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvbw5/must-gather-g64rj" event={"ID":"69b85226-16b0-4661-8bc6-7d0a0236630a","Type":"ContainerStarted","Data":"2ad6507aca8ed262e870eb4588eac6c20bec2ce05a25afd1e9a3af548dad082a"}
Sep 12 00:59:22.425378 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:22.425329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wvbw5/must-gather-g64rj" podStartSLOduration=1.641226623 podStartE2EDuration="2.425313913s" totalImagesPullingTime="784.08729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:22.425105094 +0000 UTC m=+2565.362411614" watchObservedRunningTime="2026-09-12 00:59:22.425313913 +0000 UTC m=+2565.362620432"
Sep 12 00:59:23.712329 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:23.712293    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7tm66_be73927c-b191-4bef-8ded-c6bee22f4ae0/global-pull-secret-syncer/0.log"
Sep 12 00:59:23.798608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:23.798575    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2rthv_ce54e557-520d-4218-b42b-2571f6e04a5b/konnectivity-agent/0.log"
Sep 12 00:59:23.901895 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:23.901863    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-187.ec2.internal_7f8f2c475d91043fb3907657c68af0ed/haproxy/0.log"
Sep 12 00:59:28.408209 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:28.408147    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-h7ptz_8a79b70d-f67d-4dc1-b8e1-89c66440c9f7/manager/0.log"
Sep 12 00:59:28.434709 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:28.434684    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tf8t9_56571bde-c5fa-4aae-a2b2-67a1a5234cd5/kuadrant-console-plugin/0.log"
Sep 12 00:59:28.489355 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:28.489309    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gspqm_e8f5fadf-fd3d-4129-af60-2470675885b4/manager/0.log"
Sep 12 00:59:28.515746 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:28.515712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t5k2j_ce17ca8a-f51c-4ca3-a856-9398789d4dca/limitador/0.log"
Sep 12 00:59:28.585739 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:28.585685    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-n6q84_fd3180a5-2d82-4349-862b-ce5789e11ca1/manager/0.log"
Sep 12 00:59:30.197135 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.197088    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/alertmanager/0.log"
Sep 12 00:59:30.219537 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.219506    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/config-reloader/0.log"
Sep 12 00:59:30.247410 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.247382    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/kube-rbac-proxy-web/0.log"
Sep 12 00:59:30.271020 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.270988    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/kube-rbac-proxy/0.log"
Sep 12 00:59:30.295039 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.295001    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/kube-rbac-proxy-metric/0.log"
Sep 12 00:59:30.314562 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.314536    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/prom-label-proxy/0.log"
Sep 12 00:59:30.334462 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.334430    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91689675-72b3-4b55-a95b-675c191cf870/init-config-reloader/0.log"
Sep 12 00:59:30.716194 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.716151    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w99cd_d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea/node-exporter/0.log"
Sep 12 00:59:30.737242 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.737216    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w99cd_d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea/kube-rbac-proxy/0.log"
Sep 12 00:59:30.765091 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:30.765052    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w99cd_d511e8f7-5ec7-470c-b7ff-c66ed1a5f1ea/init-textfile/0.log"
Sep 12 00:59:31.210017 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:31.209989    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-589dd97556-tc7hb_94edbbbf-ee22-4f2e-9444-634d7faec842/telemeter-client/0.log"
Sep 12 00:59:31.330600 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:31.330555    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-589dd97556-tc7hb_94edbbbf-ee22-4f2e-9444-634d7faec842/reload/0.log"
Sep 12 00:59:31.397433 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:31.397349    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-589dd97556-tc7hb_94edbbbf-ee22-4f2e-9444-634d7faec842/kube-rbac-proxy/0.log"
Sep 12 00:59:32.989522 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:32.989462    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-xdfwv_6aba9d5b-aa98-4295-a190-707f83930beb/networking-console-plugin/0.log"
Sep 12 00:59:33.594528 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.594496    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/1.log"
Sep 12 00:59:33.600677 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.600645    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-b5l7l_4947b843-449b-4830-a0fb-66cbe4077a4e/console-operator/2.log"
Sep 12 00:59:33.794950 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.794917    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"]
Sep 12 00:59:33.798796 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.798763    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.813372 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.813346    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"]
Sep 12 00:59:33.882921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.882821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-sys\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.882921 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.882899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-podres\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.883151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.882936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-lib-modules\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.883151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.882961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbck\" (UniqueName: \"kubernetes.io/projected/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-kube-api-access-7pbck\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.883151 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.882996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-proc\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.983795 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.983750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-podres\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984070 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-podres\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984209 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-lib-modules\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984308 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-lib-modules\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984379 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbck\" (UniqueName: \"kubernetes.io/projected/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-kube-api-access-7pbck\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984431 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-proc\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984510 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-sys\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.984625 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.984608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-sys\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:33.985231 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:33.985206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-proc\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:34.007176 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:34.007150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbck\" (UniqueName: \"kubernetes.io/projected/307fa3bc-ee1d-4558-a6a9-a730fb1350eb-kube-api-access-7pbck\") pod \"perf-node-gather-daemonset-rrwwp\" (UID: \"307fa3bc-ee1d-4558-a6a9-a730fb1350eb\") " pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:34.113659 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:34.113616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:34.405027 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:34.404990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"]
Sep 12 00:59:34.408357 ip-10-0-140-187 kubenswrapper[2562]: W0912 00:59:34.408323    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod307fa3bc_ee1d_4558_a6a9_a730fb1350eb.slice/crio-839439ef2bb5c76ead0dfc299a0921b27112fb4902f2780d67600098dc2eceb8 WatchSource:0}: Error finding container 839439ef2bb5c76ead0dfc299a0921b27112fb4902f2780d67600098dc2eceb8: Status 404 returned error can't find the container with id 839439ef2bb5c76ead0dfc299a0921b27112fb4902f2780d67600098dc2eceb8
Sep 12 00:59:34.499671 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:34.499628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp" event={"ID":"307fa3bc-ee1d-4558-a6a9-a730fb1350eb","Type":"ContainerStarted","Data":"839439ef2bb5c76ead0dfc299a0921b27112fb4902f2780d67600098dc2eceb8"}
Sep 12 00:59:35.510946 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:35.510901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp" event={"ID":"307fa3bc-ee1d-4558-a6a9-a730fb1350eb","Type":"ContainerStarted","Data":"bfe3f73fffde1b6d4d27173a3bbd8b2974afb38f2355394aac56f126a46ed333"}
Sep 12 00:59:35.511383 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:35.511105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:35.527401 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:35.527344    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp" podStartSLOduration=2.527326192 podStartE2EDuration="2.527326192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:35.527137721 +0000 UTC m=+2578.464444242" watchObservedRunningTime="2026-09-12 00:59:35.527326192 +0000 UTC m=+2578.464632713"
Sep 12 00:59:36.100676 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:36.100620    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cpf2k_9ba59c17-33b2-4642-a51c-e92070a474d0/dns/0.log"
Sep 12 00:59:36.123319 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:36.123288    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cpf2k_9ba59c17-33b2-4642-a51c-e92070a474d0/kube-rbac-proxy/0.log"
Sep 12 00:59:36.291783 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:36.291758    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ttcd9_a1a021a5-f8de-48be-8456-1c71ef09a626/dns-node-resolver/0.log"
Sep 12 00:59:36.807858 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:36.807831    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fc78bdd69-nrwsg_a46fb3c0-061d-49b1-90e6-5626f7a689e3/registry/0.log"
Sep 12 00:59:36.819362 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:36.819336    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fc78bdd69-nrwsg_a46fb3c0-061d-49b1-90e6-5626f7a689e3/registry/1.log"
Sep 12 00:59:36.843814 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:36.843789    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gxj9b_4353938a-ab7e-4fd7-b93c-2336a53ff5f9/node-ca/0.log"
Sep 12 00:59:38.018684 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:38.018647    2562 scope.go:117] "RemoveContainer" containerID="35234abdda838150961cd8e497b290de9bfbb4b2ae7c6fcb9c74b9dded720a64"
Sep 12 00:59:38.028608 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:38.028588    2562 scope.go:117] "RemoveContainer" containerID="72d99c0b0e1c27d0935701c4c9152415e562381210b9e313f9f98cf361a9cb45"
Sep 12 00:59:38.037860 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:38.037831    2562 scope.go:117] "RemoveContainer" containerID="673553a4690e77e32169ebb853a8b834f8a6850a1f97d11aafbad9dc60c25319"
Sep 12 00:59:39.247165 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:39.247139    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5hxx_9e8227bd-18e8-4220-82fd-2e9878a7aecb/serve-healthcheck-canary/0.log"
Sep 12 00:59:39.781027 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:39.780982    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qd42h_b3562c34-1caf-42eb-9b2a-56abc7092a19/insights-operator/0.log"
Sep 12 00:59:39.781237 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:39.781183    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qd42h_b3562c34-1caf-42eb-9b2a-56abc7092a19/insights-operator/1.log"
Sep 12 00:59:39.868022 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:39.867997    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l64ss_d6ae1947-3afa-47ca-913f-9744a21324f0/kube-rbac-proxy/0.log"
Sep 12 00:59:39.891092 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:39.891051    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l64ss_d6ae1947-3afa-47ca-913f-9744a21324f0/exporter/0.log"
Sep 12 00:59:39.911055 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:39.911030    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l64ss_d6ae1947-3afa-47ca-913f-9744a21324f0/extractor/0.log"
Sep 12 00:59:41.528738 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:41.528703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wvbw5/perf-node-gather-daemonset-rrwwp"
Sep 12 00:59:42.626381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:42.626348    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446f459d6-p9cwf_c4e6a6c5-8374-4047-8472-7372b0aed510/manager/0.log"
Sep 12 00:59:42.643342 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:42.643279    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8446f459d6-p9cwf_c4e6a6c5-8374-4047-8472-7372b0aed510/copy-manifests/0.log"
Sep 12 00:59:44.154939 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:44.154912    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b5cf77b9-n9jgm_0ef40f12-355a-4fc0-9158-afba7b521387/manager/0.log"
Sep 12 00:59:44.204047 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:44.204017    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-znjlz_6684d846-3d3a-4cee-baf5-c597a1f2e9b0/openshift-lws-operator/0.log"
Sep 12 00:59:49.194782 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:49.194752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-xplln_0e565eed-f414-408a-b225-3a1c948d9826/migrator/0.log"
Sep 12 00:59:49.214186 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:49.214157    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-xplln_0e565eed-f414-408a-b225-3a1c948d9826/graceful-termination/0.log"
Sep 12 00:59:49.582614 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:49.582583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-bjdzv_87f8b85f-b68a-47c2-abbd-091e63c3f3d1/kube-storage-version-migrator-operator/1.log"
Sep 12 00:59:49.583660 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:49.583634    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-bjdzv_87f8b85f-b68a-47c2-abbd-091e63c3f3d1/kube-storage-version-migrator-operator/0.log"
Sep 12 00:59:50.794706 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.794635    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/kube-multus-additional-cni-plugins/0.log"
Sep 12 00:59:50.816160 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.816131    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/egress-router-binary-copy/0.log"
Sep 12 00:59:50.837011 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.836974    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/cni-plugins/0.log"
Sep 12 00:59:50.858381 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.858343    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/bond-cni-plugin/0.log"
Sep 12 00:59:50.881442 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.881395    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/routeoverride-cni/0.log"
Sep 12 00:59:50.900803 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.900781    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/whereabouts-cni-bincopy/0.log"
Sep 12 00:59:50.922159 ip-10-0-140-187 kubenswrapper[2562]: I0912 00:59:50.922128    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mrzr4_592b7d6c-e39d-49c4-8d7b-4d4e15ae9b2d/whereabouts-cni/0.log"