Sep 14 10:59:04.018429 ip-10-0-143-154 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 14 10:59:04.018444 ip-10-0-143-154 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 14 10:59:04.018454 ip-10-0-143-154 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 14 10:59:04.018685 ip-10-0-143-154 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 14 10:59:14.228891 ip-10-0-143-154 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 14 10:59:14.228905 ip-10-0-143-154 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 38317899a8df442286a3b319904d65c0 --
Sep 14 11:00:50.188849 ip-10-0-143-154 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 11:00:50.581441 ip-10-0-143-154 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 11:00:50.581441 ip-10-0-143-154 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 11:00:50.581441 ip-10-0-143-154 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 11:00:50.581441 ip-10-0-143-154 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 11:00:50.581441 ip-10-0-143-154 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 11:00:50.583965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.583883    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 11:00:50.590174 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590150    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 11:00:50.590174 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590170    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 11:00:50.590174 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590174    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 11:00:50.590174 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590180    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590184    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590187    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590191    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590194    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590197    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590200    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590204    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590209    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590212    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590215    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590219    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590221    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590224    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590227    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590230    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590233    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590235    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590238    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590241    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 11:00:50.590349 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590248    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590251    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590253    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590256    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590258    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590261    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590264    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590267    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590269    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590272    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590274    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590277    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590280    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590283    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590288    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590291    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590294    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590297    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590299    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 11:00:50.590849 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590303    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590308    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590311    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590314    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590317    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590320    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590323    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590326    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590330    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590333    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590335    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590338    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590341    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590344    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590346    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590349    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590354    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590358    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590361    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590364    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 11:00:50.591318 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590367    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590370    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590372    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590375    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590378    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590380    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590384    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590387    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590390    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590392    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590395    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590397    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590400    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590403    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590405    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590408    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590410    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590413    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590415    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590418    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 11:00:50.591818 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590420    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590423    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590426    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590428    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590431    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590433    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590436    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590438    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590441    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590443    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590446    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590448    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590451    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590453    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590456    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590458    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 11:00:50.592328 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.590461    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 11:00:50.593908 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593895    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 11:00:50.593908 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593907    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593911    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593914    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593917    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593920    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593923    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593926    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593929    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593933    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593936    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593939    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593941    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593944    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593946    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593950    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593954    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593957    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593960    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593963    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 11:00:50.593971 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593965    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593968    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593970    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593973    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593976    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593979    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593982    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593985    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593988    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593991    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593993    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593996    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.593998    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594001    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594003    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594006    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594008    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594011    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594013    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594016    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 11:00:50.594437 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594018    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594021    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594023    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594026    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594029    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594031    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594034    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594036    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594039    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594041    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594044    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594047    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594050    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594052    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594055    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594058    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594061    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594063    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594066    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 11:00:50.594954 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594070    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594075    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594078    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594081    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594083    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594086    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594088    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594090    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594093    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594095    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594098    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594101    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594103    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594106    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594108    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594110    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594113    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594116    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594118    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594121    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 11:00:50.595415 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594124    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594126    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594129    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594132    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594135    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594137    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594140    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594143    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594146    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594148    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594151    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594153    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594156    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594159    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594162    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594165    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594167    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594170    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594172    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594175    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 11:00:50.595929 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594539    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594544    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594562    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594566    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594569    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594572    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594575    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594578    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594581    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594584    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594587    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594589    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594592    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594595    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594597    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594599    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594602    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594605    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594608    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594611    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 11:00:50.596419 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594613    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594616    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594619    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594621    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594624    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594627    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594629    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594632    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594634    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594637    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594639    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594642    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594644    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594647    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594650    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594652    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594654    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594657    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594659    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594662    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 11:00:50.596925 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594664    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594667    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594669    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594671    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594674    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594677    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594680    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594682    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594685    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594688    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594691    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594694    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594696    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594699    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594701    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594704    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594706    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594709    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594711    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594714    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 11:00:50.597420 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594717    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594719    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594722    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594724    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594726    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594729    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594731    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594734    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594737    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594739    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594741    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594744    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594748    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594751    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594753    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594756    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594758    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594761    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594763    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 11:00:50.597974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594766    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594769    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594772    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594775    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594778    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594781    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594783    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594787    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594789    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594792    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594795    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594797    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594800    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594804    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594807    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594810    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594813    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594815    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594818    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.594820    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 11:00:50.598433 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594893    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594937    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594946    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594955    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594959    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594963    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594967    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594971    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594974    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594978    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594981    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594985    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594988    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594991    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594994    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.594998    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595001    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595004    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595008    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595011    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595014    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595017    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595023    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595026    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 11:00:50.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595029    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595032    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595036    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595039    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595042    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595045    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595048    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595051    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595055    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595058    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595061    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595063    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595066    2563 flags.go:64] FLAG: --enable-server="true"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595071    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595076    2563 flags.go:64] FLAG: --event-burst="100"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595079    2563 flags.go:64] FLAG: --event-qps="50"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595082    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595085    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595087    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595091    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595094    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595097    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595100    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595102    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595105    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 11:00:50.599514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595109    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595112    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595114    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595117    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595120    2563 flags.go:64] FLAG: --feature-gates=""
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595125    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595128    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595131    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595134    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595137    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595140    2563 flags.go:64] FLAG: --help="false"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595143    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595146    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595149    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595152    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595156    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595159    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595162    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595164    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595167    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595171    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595174    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595177    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595180    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 14 11:00:50.600132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595183    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595186    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595189    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595192    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595194    2563 flags.go:64] FLAG: --lock-file=""
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595197    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595200    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595203    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595208    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595212    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595215    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595217    2563 flags.go:64] FLAG: --logging-format="text"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595220    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595225    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595228    2563 flags.go:64] FLAG: --manifest-url=""
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595231    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595235    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595238    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595241    2563 flags.go:64] FLAG: --max-pods="110"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595244    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595247    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595250    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595253    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595256    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595259    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 11:00:50.600730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595261    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595268    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595272    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595276    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595279    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595282    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595287    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595290    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595293    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595296    2563 flags.go:64] FLAG: --port="10250"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595299    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595302    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07bbd4bf1e06e4986"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595305    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595308    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595311    2563 flags.go:64] FLAG: --register-node="true"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595314    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595318    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595322    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595325    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595327    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595332    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595335    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595338    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595341    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595344    2563 flags.go:64] FLAG: --runonce="false"
Sep 14 11:00:50.601380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595347    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595350    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595353    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595356    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595359    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595362    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595366    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595369    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595372    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595375    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595378    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595382    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595385    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595388    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595391    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595396    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595399    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595403    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595406    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595409    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595412    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595415    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595418    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595421    2563 flags.go:64] FLAG: --v="2"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595425    2563 flags.go:64] FLAG: --version="false"
Sep 14 11:00:50.602024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595430    2563 flags.go:64] FLAG: --vmodule=""
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595433    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.595437    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595547    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595563    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595566    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595571    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595574    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595577    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595580    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595583    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595587    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595590    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595592    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595595    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595597    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595600    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595604    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595608    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595610    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 11:00:50.602665 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595613    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595616    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595619    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595621    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595624    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595626    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595629    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595632    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595635    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595638    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595641    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595645    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595648    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595652    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595655    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595658    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595661    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595663    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595666    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 11:00:50.603478 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595669    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595671    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595674    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595676    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595679    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595681    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595684    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595686    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595689    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595691    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595694    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595697    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595700    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595703    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595705    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595708    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595710    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595713    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595716    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595719    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595721    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 11:00:50.604308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595724    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595726    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595729    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595732    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595734    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595737    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595740    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595743    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595746    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595748    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595751    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595753    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595756    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595759    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595761    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595772    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595774    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595777    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595780    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595782    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 11:00:50.605220 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595785    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595787    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595792    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595795    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595798    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595801    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595803    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595806    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595809    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595812    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595815    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595818    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595821    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595825    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595827    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595831    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595833    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595836    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595838    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 11:00:50.605745 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595841    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595844    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.595847    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.596462    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.604947    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.604966    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.605908    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.605924    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606009    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606018    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606025    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606031    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606035    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606040    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 11:00:50.606534 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606044    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606049    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606053    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606058    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606062    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606067    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606071    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606075    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606080    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606084    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606088    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606094    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606098    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606102    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606107    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606113    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606119    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606125    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606130    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606134    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 11:00:50.607178 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606139    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606144    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606148    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606154    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606158    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606162    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606166    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606171    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606175    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606179    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606183    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606187    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606192    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606196    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606201    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606205    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606209    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606213    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606217    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606221    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 11:00:50.607776 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606225    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606230    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606234    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606238    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606243    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606247    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606251    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606256    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606260    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606264    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606269    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606273    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606277    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606281    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606285    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606289    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606295    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606299    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606303    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 11:00:50.608520 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606307    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606311    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606316    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606319    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606323    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606327    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606332    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606336    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606340    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606345    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606349    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606353    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606357    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606361    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606365    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606369    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606374    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606378    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606382    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606386    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 11:00:50.609350 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606391    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606395    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606399    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606403    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606407    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606411    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606415    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606419    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606424    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606428    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606435    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606441    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606446    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606450    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 11:00:50.609902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.606458    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606650    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606660    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606665    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606669    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606673    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606678    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606682    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606687    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606691    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606695    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606699    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606703    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606707    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606712    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606716    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606720    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606724    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606728    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606733    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 11:00:50.610356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606737    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606741    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606746    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606750    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606754    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606758    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606762    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606767    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606771    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606776    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606781    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606785    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606789    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606793    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606797    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606801    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606805    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606810    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606815    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 11:00:50.610892 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606819    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606823    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606827    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606834    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606840    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606845    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606850    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606854    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606859    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606863    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606867    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606871    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606875    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606879    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606884    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606888    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606892    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606896    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606900    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 11:00:50.611409 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606904    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606909    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606913    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606918    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606923    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606927    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606931    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606935    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606939    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606943    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606947    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606952    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606956    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606960    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606964    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606968    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606972    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606976    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606980    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606984    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 11:00:50.611895 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606988    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606992    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.606996    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607001    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607005    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607010    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607015    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607020    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607024    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607029    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607033    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607037    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607041    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607045    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607050    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607056    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607062    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607068    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607073    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 11:00:50.612449 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607078    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607081    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:50.607085    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.607094    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.607239    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.610177    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.611295    2563 server.go:1019] "Starting client certificate rotation"
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.611402    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 11:00:50.612934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.611443    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 11:00:50.615136 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.615119    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 11:00:50.621864 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.621842    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 11:00:50.636313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.636298    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 14 11:00:50.639648 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.639629    2563 log.go:25] "Validated CRI v1 image API"
Sep 14 11:00:50.640470 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.640457    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 11:00:50.641145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.641131    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 11:00:50.643959 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.643935    2563 fs.go:135] Filesystem UUIDs: map[25741e9b-6026-4d97-97a1-cf663339ede6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9eefa3df-833a-4972-ac34-1d24b4d74839:/dev/nvme0n1p3]
Sep 14 11:00:50.644054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.643959    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 11:00:50.649361 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.649253    2563 manager.go:216] Machine: {Timestamp:2026-09-14 11:00:50.647795345 +0000 UTC m=+0.366186468 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096181 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24eb05de5ed95f426213af7bab2e24 SystemUUID:ec24eb05-de5e-d95f-4262-13af7bab2e24 BootID:38317899-a8df-4422-86a3-b319904d65c0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:d2:dc:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:d2:dc:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:ed:b3:80:5c:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 11:00:50.649361 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.649349    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 11:00:50.649522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.649498    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 11:00:50.651525 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.651498    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 11:00:50.651666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.651527    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-154.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 11:00:50.653097 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.653087    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 11:00:50.653139 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.653099    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 11:00:50.653139 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.653107    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 11:00:50.653979 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.653968    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 11:00:50.655260 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.655251    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 11:00:50.660042 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660030    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 11:00:50.660161 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660151    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 11:00:50.660202 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660180    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 11:00:50.660202 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660198    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 11:00:50.660266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660212    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 11:00:50.660266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660223    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 14 11:00:50.660266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660238    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 11:00:50.660646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660634    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 11:00:50.660689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.660660    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 11:00:50.662825 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.662810    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 11:00:50.665676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.665663    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 11:00:50.665717 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.665692    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 11:00:50.668470 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668455    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 11:00:50.668532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668481    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 11:00:50.668532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668491    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 11:00:50.668532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668499    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 11:00:50.668532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668512    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 11:00:50.668532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668520    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 11:00:50.668688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668539    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 11:00:50.668688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668547    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 11:00:50.668688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668571    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 11:00:50.668688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668580    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 11:00:50.668688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668591    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 11:00:50.668688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.668599    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 11:00:50.669234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.669216    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbppp"
Sep 14 11:00:50.669405 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.669393    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 11:00:50.669436 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.669406    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 11:00:50.671418 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.671388    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 11:00:50.671587 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.671517    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 11:00:50.673859 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.673844    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-154.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 11:00:50.674434 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.674413    2563 server.go:1268] "Started kubelet"
Sep 14 11:00:50.674533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.674463    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 11:00:50.674921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.674876    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 11:00:50.674988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.674943    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 11:00:50.675242 ip-10-0-143-154 systemd[1]: Started Kubernetes Kubelet.
Sep 14 11:00:50.676297 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.676275    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbppp"
Sep 14 11:00:50.676657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.676631    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 11:00:50.676738 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.676699    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 11:00:50.683224 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.682062    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-154.ec2.internal.18d52af32156a117  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-154.ec2.internal,UID:ip-10-0-143-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-154.ec2.internal,},FirstTimestamp:2026-09-14 11:00:50.674385175 +0000 UTC m=+0.392776302,LastTimestamp:2026-09-14 11:00:50.674385175 +0000 UTC m=+0.392776302,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-154.ec2.internal,}"
Sep 14 11:00:50.683648 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.683630    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 11:00:50.686454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.686436    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 11:00:50.687089 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687072    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 11:00:50.687229 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.687212    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:50.687876 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687859    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 11:00:50.687876 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687860    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 11:00:50.688006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687885    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 11:00:50.688006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687916    2563 factory.go:55] Registering systemd factory
Sep 14 11:00:50.688006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687932    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 14 11:00:50.688006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687963    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 11:00:50.688006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.687974    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 11:00:50.688229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.688216    2563 factory.go:153] Registering CRI-O factory
Sep 14 11:00:50.688259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.688234    2563 factory.go:223] Registration of the crio container factory successfully
Sep 14 11:00:50.688297 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.688286    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 11:00:50.688330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.688312    2563 factory.go:103] Registering Raw factory
Sep 14 11:00:50.688330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.688328    2563 manager.go:1196] Started watching for new ooms in manager
Sep 14 11:00:50.688732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.688721    2563 manager.go:349] Starting recovery of all containers
Sep 14 11:00:50.696364 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.696204    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 11:00:50.700330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.700306    2563 manager.go:354] Recovery completed
Sep 14 11:00:50.700837 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.700789    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-154.ec2.internal\" not found" node="ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.700894 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.700881    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 11:00:50.707006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.706990    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.708846 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.708831    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.708913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.708859    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.708913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.708869    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.709446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.709432    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 11:00:50.709446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.709444    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 11:00:50.709577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.709464    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 11:00:50.709577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.709447    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.710171 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.710150    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.710229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.710181    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.710229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.710191    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.711621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.711608    2563 policy_none.go:49] "None policy: Start"
Sep 14 11:00:50.711674 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.711624    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 11:00:50.711674 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.711637    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 11:00:50.713644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.713627    2563 policy_none.go:47] "Start"
Sep 14 11:00:50.759160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759145    2563 manager.go:337] "Starting Device Plugin manager"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.759178    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759187    2563 server.go:87] "Starting device plugin registration server"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759447    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759457    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759543    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759630    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.759638    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.760066    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 11:00:50.771388 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.760105    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:50.815963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.815941    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 11:00:50.816048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.815983    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 11:00:50.816048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.816003    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 11:00:50.816048 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.816039    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 11:00:50.818578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.818546    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 11:00:50.859857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.859792    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.861116 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.861104    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.861184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.861128    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.861184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.861138    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.861184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.861161    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.869413 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.869398    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.869476 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.869419    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-154.ec2.internal\": node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:50.887157 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.887136    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:50.917044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.917013    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"]
Sep 14 11:00:50.917128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.917091    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.918383 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.918368    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.918449 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.918395    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.918449 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.918407    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.920643 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.920630    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.920772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.920747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.920818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.920785    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.921289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.921276    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.921289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.921284    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.921416 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.921302    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.921416 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.921306    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.921416 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.921315    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.921416 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.921321    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.923512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.923496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.923602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.923525    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 11:00:50.924233 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.924221    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 11:00:50.924303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.924242    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 11:00:50.924303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.924255    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeHasSufficientPID"
Sep 14 11:00:50.952085 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.952063    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-154.ec2.internal\" not found" node="ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.956321 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.956304    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-154.ec2.internal\" not found" node="ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.988009 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:50.987987    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:50.990170 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.990155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45284353ad5e38e301b33a8146d7b581-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal\" (UID: \"45284353ad5e38e301b33a8146d7b581\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.990225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.990177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45284353ad5e38e301b33a8146d7b581-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal\" (UID: \"45284353ad5e38e301b33a8146d7b581\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:50.990225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:50.990194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed43c03ec92d5b25a955c89d154a23b6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-154.ec2.internal\" (UID: \"ed43c03ec92d5b25a955c89d154a23b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.088923 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:51.088903    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:51.091090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.091074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45284353ad5e38e301b33a8146d7b581-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal\" (UID: \"45284353ad5e38e301b33a8146d7b581\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.091141 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.091101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45284353ad5e38e301b33a8146d7b581-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal\" (UID: \"45284353ad5e38e301b33a8146d7b581\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.091141 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.091119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed43c03ec92d5b25a955c89d154a23b6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-154.ec2.internal\" (UID: \"ed43c03ec92d5b25a955c89d154a23b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.091203 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.091178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed43c03ec92d5b25a955c89d154a23b6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-154.ec2.internal\" (UID: \"ed43c03ec92d5b25a955c89d154a23b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.091247 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.091235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45284353ad5e38e301b33a8146d7b581-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal\" (UID: \"45284353ad5e38e301b33a8146d7b581\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.091283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.091259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45284353ad5e38e301b33a8146d7b581-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal\" (UID: \"45284353ad5e38e301b33a8146d7b581\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.189633 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:51.189565    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:51.254894 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.254875    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.259771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:51.259598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:51.290382 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:51.290358    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:52.486763 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:52.486729    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:52.587220 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:52.587167    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:52.687813 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:52.687791    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:52.707074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.707053    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 11:00:52.707196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.707180    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 11:00:52.707236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.707207    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 11:00:52.779424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.779380    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 10:55:51 +0000 UTC" deadline="2028-05-17 23:22:37.274516666 +0000 UTC"
Sep 14 11:00:52.779424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.779418    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14676h21m44.495101927s"
Sep 14 11:00:52.788291 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:52.788272    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:52.888972 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:52.888895    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-154.ec2.internal\" not found"
Sep 14 11:00:52.910772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.910751    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 11:00:52.984035 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.984013    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:52.996411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.996382    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 11:00:52.997801 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:52.997786    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal"
Sep 14 11:00:53.006352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.006332    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 11:00:53.049611 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.049585    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 11:00:53.087657 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:53.087628    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45284353ad5e38e301b33a8146d7b581.slice/crio-826eebd68596c778be5d35c31e7b7cae582269c898c1e15a96e291c1151532d8 WatchSource:0}: Error finding container 826eebd68596c778be5d35c31e7b7cae582269c898c1e15a96e291c1151532d8: Status 404 returned error can't find the container with id 826eebd68596c778be5d35c31e7b7cae582269c898c1e15a96e291c1151532d8
Sep 14 11:00:53.089141 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:53.089120    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded43c03ec92d5b25a955c89d154a23b6.slice/crio-8a94a41c0021d4a3d4c1416927930dc1a296cbf56d307aa191e9466883cfcfef WatchSource:0}: Error finding container 8a94a41c0021d4a3d4c1416927930dc1a296cbf56d307aa191e9466883cfcfef: Status 404 returned error can't find the container with id 8a94a41c0021d4a3d4c1416927930dc1a296cbf56d307aa191e9466883cfcfef
Sep 14 11:00:53.094461 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.094440    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:00:53.756657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.756624    2563 apiserver.go:52] "Watching apiserver"
Sep 14 11:00:53.765378 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.765354    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 11:00:53.766288 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.766266    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x7sw9","kube-system/konnectivity-agent-4lglg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8","openshift-cluster-node-tuning-operator/tuned-vcv6v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal","openshift-multus/multus-additional-cni-plugins-fqzz9","openshift-network-diagnostics/network-check-target-jrkpt","openshift-ovn-kubernetes/ovnkube-node-h28cz","kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal","openshift-dns/node-resolver-d8dg5","openshift-image-registry/node-ca-7psjm","openshift-multus/multus-jbccv","openshift-multus/network-metrics-daemon-jplcp","openshift-network-operator/iptables-alerter-sr4fm"]
Sep 14 11:00:53.768724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.768703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.770916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.770893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.771312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.771291    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.771312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.771306    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.771451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.771330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 11:00:53.771600 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.771582    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 11:00:53.772149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.772127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 11:00:53.772266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.772128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 11:00:53.772266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.772133    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcxrq\""
Sep 14 11:00:53.773255 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.773125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.773255 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.773190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.773518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.773258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 11:00:53.773518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.773441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:53.773518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.773474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g28p\""
Sep 14 11:00:53.773685 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.773536    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:00:53.775352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.775335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.777728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.777709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.780092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.779889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.780092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.779929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.780092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.779944    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 11:00:53.780092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.779975    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.780304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.780184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.780304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.780190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.780304 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.780231    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:00:53.780516 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.780499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dl6mn\""
Sep 14 11:00:53.780589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.780560    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 11:00:53.780739 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.780715    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 11:00:53.780817 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.780806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hb8d\""
Sep 14 11:00:53.782373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.782357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.782701 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.782528    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 11:00:53.784581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.784556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.787182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.787162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:53.787589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.787573    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.787752 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.787722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 11:00:53.787828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.787753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.787881 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.787824    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.788107 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.788081    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.788174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.788085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zmppr\""
Sep 14 11:00:53.788273 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.788256    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t76lc\""
Sep 14 11:00:53.789699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.789640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 11:00:53.789699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.789646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.789943 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.789921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 11:00:53.790026 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.789936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 11:00:53.790283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.790264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c74t\""
Sep 14 11:00:53.791928 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.791910    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g9rkp\""
Sep 14 11:00:53.792116 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.791956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 11:00:53.792475 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.792410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:53.792475 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.792466    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:00:53.794127 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-sys\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.794216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-systemd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.794216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-cni-bin\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.794216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-systemd\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.794331 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-system-cni-dir\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.794331 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9rp\" (UniqueName: \"kubernetes.io/projected/0983107e-95db-4d04-9e20-d4e4f17a3139-kube-api-access-bd9rp\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.794476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.794476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.794476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.794476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n6sm\" (UniqueName: \"kubernetes.io/projected/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-kube-api-access-9n6sm\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.794476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-run-netns\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.794476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-socket-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-modprobe-d\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysconfig\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-hosts-file\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82f1808b-0670-4a82-8c87-39180882715e-kubelet-config\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-run\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.794735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-tuned\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sn4x\" (UniqueName: \"kubernetes.io/projected/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-kube-api-access-5sn4x\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-slash\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-ovn\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovnkube-script-lib\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82f1808b-0670-4a82-8c87-39180882715e-dbus\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.794961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-sys-fs\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysctl-d\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-var-lib-kubelet\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-tmp-dir\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.795145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-etc-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-host\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-env-overrides\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fwd\" (UniqueName: \"kubernetes.io/projected/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-kube-api-access-69fwd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-lib-modules\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-node-log\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-kubernetes\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.795790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cda162ae-636e-43a3-ab3e-5dafb5351179-iptables-alerter-script\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.796158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rsh4\" (UniqueName: \"kubernetes.io/projected/98fe2c4f-cf79-4633-8472-41a9f27c41fe-kube-api-access-8rsh4\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.796158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-kubelet\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-log-socket\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovnkube-config\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-device-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.796158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.795986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98fe2c4f-cf79-4633-8472-41a9f27c41fe-tmp\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cnibin\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-systemd-units\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-cni-netd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovn-node-metrics-cert\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda162ae-636e-43a3-ab3e-5dafb5351179-host-slash\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt25\" (UniqueName: \"kubernetes.io/projected/cda162ae-636e-43a3-ab3e-5dafb5351179-kube-api-access-btt25\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-var-lib-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-registration-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.796531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 11:00:53.796961 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysctl-conf\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.796961 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.796961 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.797112 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.796998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 11:00:53.797244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.797228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svjvk\""
Sep 14 11:00:53.801622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.801595    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 11:00:53.825530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.825508    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z7gkv"
Sep 14 11:00:53.834678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.834604    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z7gkv"
Sep 14 11:00:53.837451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.837434    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 11:00:53.885058 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.885031    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 11:00:53.897175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-sys\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.897175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-systemd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.897371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-cni-bin\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.897371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-cnibin\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.897371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-systemd\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.897371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-sys\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.897371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-systemd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-system-cni-dir\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-system-cni-dir\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9rp\" (UniqueName: \"kubernetes.io/projected/0983107e-95db-4d04-9e20-d4e4f17a3139-kube-api-access-bd9rp\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-cni-bin\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-socket-dir-parent\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-systemd\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-cni-multus\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6sm\" (UniqueName: \"kubernetes.io/projected/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-kube-api-access-9n6sm\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-run-netns\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.897733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-kubelet\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-socket-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-k8s-cni-cncf-io\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-modprobe-d\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysconfig\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-hosts-file\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82f1808b-0670-4a82-8c87-39180882715e-kubelet-config\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-socket-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-netns\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-run-netns\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-run\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysconfig\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-tuned\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82f1808b-0670-4a82-8c87-39180882715e-kubelet-config\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sn4x\" (UniqueName: \"kubernetes.io/projected/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-kube-api-access-5sn4x\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.898337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-slash\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-hosts-file\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-ovn\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.897838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovnkube-script-lib\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-slash\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82f1808b-0670-4a82-8c87-39180882715e-dbus\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-sys-fs\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82f1808b-0670-4a82-8c87-39180882715e-dbus\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4dm\" (UniqueName: \"kubernetes.io/projected/067aa408-be71-402b-aada-cd64b57c95f0-kube-api-access-mr4dm\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-run-ovn\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-modprobe-d\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/422e099d-bace-421a-a944-8f29a52a5ff5-agent-certs\") pod \"konnectivity-agent-4lglg\" (UID: \"422e099d-bace-421a-a944-8f29a52a5ff5\") " pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-system-cni-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.899147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-etc-kubernetes\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysctl-d\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-run\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-var-lib-kubelet\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-tmp-dir\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-etc-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.898554    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.898612    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:00:54.398593286 +0000 UTC m=+3.021146480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-cni-bin\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-host\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-env-overrides\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fwd\" (UniqueName: \"kubernetes.io/projected/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-kube-api-access-69fwd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898699    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 11:00:53.899969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.898945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-sys-fs\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-var-lib-kubelet\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysctl-d\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a6d424e-e203-4fae-9ff5-db0982a2f37b-serviceca\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmdk\" (UniqueName: \"kubernetes.io/projected/0a6d424e-e203-4fae-9ff5-db0982a2f37b-kube-api-access-xwmdk\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-multus-certs\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-lib-modules\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-node-log\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-etc-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-host\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovnkube-script-lib\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-tmp-dir\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-lib-modules\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-node-log\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.900807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-env-overrides\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-hostroot\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-kubernetes\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.899976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cda162ae-636e-43a3-ab3e-5dafb5351179-iptables-alerter-script\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rsh4\" (UniqueName: \"kubernetes.io/projected/98fe2c4f-cf79-4633-8472-41a9f27c41fe-kube-api-access-8rsh4\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-kubelet\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-log-socket\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-kubernetes\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a6d424e-e203-4fae-9ff5-db0982a2f37b-host\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovnkube-config\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-device-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5jp\" (UniqueName: \"kubernetes.io/projected/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-kube-api-access-jm5jp\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98fe2c4f-cf79-4633-8472-41a9f27c41fe-tmp\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cnibin\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-systemd-units\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-cni-netd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovn-node-metrics-cert\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.901597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda162ae-636e-43a3-ab3e-5dafb5351179-host-slash\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt25\" (UniqueName: \"kubernetes.io/projected/cda162ae-636e-43a3-ab3e-5dafb5351179-kube-api-access-btt25\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-cni-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-var-lib-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-registration-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-cni-binary-copy\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-conf-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-daemon-config\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysctl-conf\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/422e099d-bace-421a-a944-8f29a52a5ff5-konnectivity-ca\") pod \"konnectivity-agent-4lglg\" (UID: \"422e099d-bace-421a-a944-8f29a52a5ff5\") " pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-cnibin\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cda162ae-636e-43a3-ab3e-5dafb5351179-iptables-alerter-script\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-cni-netd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-systemd-units\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.902417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-kubelet\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-log-socket\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.900959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-device-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda162ae-636e-43a3-ab3e-5dafb5351179-host-slash\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-var-lib-openvswitch\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0983107e-95db-4d04-9e20-d4e4f17a3139-registration-dir\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-sysctl-conf\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovnkube-config\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.901380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.903237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.903070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98fe2c4f-cf79-4633-8472-41a9f27c41fe-etc-tuned\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.903713 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.903241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-ovn-node-metrics-cert\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.903777 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.903755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98fe2c4f-cf79-4633-8472-41a9f27c41fe-tmp\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.909636 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.909612    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:00:53.909751 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.909641    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:00:53.909751 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.909654    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:53.909751 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:53.909714    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:00:54.40969716 +0000 UTC m=+3.032250371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:53.909910 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.909824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6sm\" (UniqueName: \"kubernetes.io/projected/c9afc335-7d75-48cf-8e86-1e41ef2df1fa-kube-api-access-9n6sm\") pod \"multus-additional-cni-plugins-fqzz9\" (UID: \"c9afc335-7d75-48cf-8e86-1e41ef2df1fa\") " pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:53.909962 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.909947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9rp\" (UniqueName: \"kubernetes.io/projected/0983107e-95db-4d04-9e20-d4e4f17a3139-kube-api-access-bd9rp\") pod \"aws-ebs-csi-driver-node-xhzb8\" (UID: \"0983107e-95db-4d04-9e20-d4e4f17a3139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:53.910923 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.910842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fwd\" (UniqueName: \"kubernetes.io/projected/5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea-kube-api-access-69fwd\") pod \"ovnkube-node-h28cz\" (UID: \"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:53.911203 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.911182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sn4x\" (UniqueName: \"kubernetes.io/projected/2c60ef6d-94b5-4a01-afb1-99c1f6e99d41-kube-api-access-5sn4x\") pod \"node-resolver-d8dg5\" (UID: \"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41\") " pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:53.911596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.911536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rsh4\" (UniqueName: \"kubernetes.io/projected/98fe2c4f-cf79-4633-8472-41a9f27c41fe-kube-api-access-8rsh4\") pod \"tuned-vcv6v\" (UID: \"98fe2c4f-cf79-4633-8472-41a9f27c41fe\") " pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:53.911857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.911834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt25\" (UniqueName: \"kubernetes.io/projected/cda162ae-636e-43a3-ab3e-5dafb5351179-kube-api-access-btt25\") pod \"iptables-alerter-sr4fm\" (UID: \"cda162ae-636e-43a3-ab3e-5dafb5351179\") " pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:53.917490 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.917443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal" event={"ID":"ed43c03ec92d5b25a955c89d154a23b6","Type":"ContainerStarted","Data":"8a94a41c0021d4a3d4c1416927930dc1a296cbf56d307aa191e9466883cfcfef"}
Sep 14 11:00:53.919671 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:53.919646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal" event={"ID":"45284353ad5e38e301b33a8146d7b581","Type":"ContainerStarted","Data":"826eebd68596c778be5d35c31e7b7cae582269c898c1e15a96e291c1151532d8"}
Sep 14 11:00:54.001880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.001842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4dm\" (UniqueName: \"kubernetes.io/projected/067aa408-be71-402b-aada-cd64b57c95f0-kube-api-access-mr4dm\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:54.002044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.001887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/422e099d-bace-421a-a944-8f29a52a5ff5-agent-certs\") pod \"konnectivity-agent-4lglg\" (UID: \"422e099d-bace-421a-a944-8f29a52a5ff5\") " pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:54.002044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.001914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-system-cni-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.001939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-etc-kubernetes\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.001980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-cni-bin\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a6d424e-e203-4fae-9ff5-db0982a2f37b-serviceca\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.002044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmdk\" (UniqueName: \"kubernetes.io/projected/0a6d424e-e203-4fae-9ff5-db0982a2f37b-kube-api-access-xwmdk\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-multus-certs\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-hostroot\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-hostroot\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-cni-bin\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-system-cni-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-etc-kubernetes\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a6d424e-e203-4fae-9ff5-db0982a2f37b-host\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5jp\" (UniqueName: \"kubernetes.io/projected/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-kube-api-access-jm5jp\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-cni-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a6d424e-e203-4fae-9ff5-db0982a2f37b-host\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-multus-certs\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-cni-binary-copy\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-conf-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-daemon-config\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.002403    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/422e099d-bace-421a-a944-8f29a52a5ff5-konnectivity-ca\") pod \"konnectivity-agent-4lglg\" (UID: \"422e099d-bace-421a-a944-8f29a52a5ff5\") " pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-cnibin\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.002470    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:54.502450823 +0000 UTC m=+3.125004026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-cni-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-conf-dir\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-socket-dir-parent\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-cni-multus\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-cnibin\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-socket-dir-parent\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-kubelet\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-k8s-cni-cncf-io\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-netns\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-netns\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a6d424e-e203-4fae-9ff5-db0982a2f37b-serviceca\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-kubelet\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.002907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-run-k8s-cni-cncf-io\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.003603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-host-var-lib-cni-multus\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.003603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-cni-binary-copy\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.003603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.002953    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 11:00:54.003603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.003125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/422e099d-bace-421a-a944-8f29a52a5ff5-konnectivity-ca\") pod \"konnectivity-agent-4lglg\" (UID: \"422e099d-bace-421a-a944-8f29a52a5ff5\") " pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:54.003603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.003530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-multus-daemon-config\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.005352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.005331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/422e099d-bace-421a-a944-8f29a52a5ff5-agent-certs\") pod \"konnectivity-agent-4lglg\" (UID: \"422e099d-bace-421a-a944-8f29a52a5ff5\") " pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:54.015182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.015125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4dm\" (UniqueName: \"kubernetes.io/projected/067aa408-be71-402b-aada-cd64b57c95f0-kube-api-access-mr4dm\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:54.015560 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.015538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmdk\" (UniqueName: \"kubernetes.io/projected/0a6d424e-e203-4fae-9ff5-db0982a2f37b-kube-api-access-xwmdk\") pod \"node-ca-7psjm\" (UID: \"0a6d424e-e203-4fae-9ff5-db0982a2f37b\") " pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.015697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.015655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5jp\" (UniqueName: \"kubernetes.io/projected/f8aabd4d-bdfe-451b-a3e3-c5252baf74b8-kube-api-access-jm5jp\") pod \"multus-jbccv\" (UID: \"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8\") " pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.080229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.080195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:00:54.088731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.088706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sr4fm"
Sep 14 11:00:54.098516 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.098493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vcv6v"
Sep 14 11:00:54.105183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.105163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fqzz9"
Sep 14 11:00:54.113996 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.113974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8"
Sep 14 11:00:54.119785 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.119765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d8dg5"
Sep 14 11:00:54.126281 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.126263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7psjm"
Sep 14 11:00:54.132062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.132033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbccv"
Sep 14 11:00:54.137724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.137699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:00:54.404475 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.404404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:54.404606 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.404497    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:54.404606 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.404565    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:00:55.404547115 +0000 UTC m=+4.027100307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:54.504740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.504711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:54.504920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.504764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:54.504920 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.504877    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:54.504920 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.504904    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:00:54.505077 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.504935    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:00:54.505077 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.504952    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:54.505077 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.504938    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:55.50491799 +0000 UTC m=+4.127471182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:54.505077 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:54.505025    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:00:55.505005154 +0000 UTC m=+4.127558346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:54.838352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.838247    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:55:53 +0000 UTC" deadline="2028-04-21 11:13:51.765545337 +0000 UTC"
Sep 14 11:00:54.838352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:54.838299    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14040h12m56.927251772s"
Sep 14 11:00:55.000875 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.000847    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422e099d_bace_421a_a944_8f29a52a5ff5.slice/crio-de2c7399cfbe8905b518fac199a85a3e41d111cce95b46240d7bf663c5c7eaf8 WatchSource:0}: Error finding container de2c7399cfbe8905b518fac199a85a3e41d111cce95b46240d7bf663c5c7eaf8: Status 404 returned error can't find the container with id de2c7399cfbe8905b518fac199a85a3e41d111cce95b46240d7bf663c5c7eaf8
Sep 14 11:00:55.002940 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.002918    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9afc335_7d75_48cf_8e86_1e41ef2df1fa.slice/crio-fcff4cfd1863d673f74002561cf0d11d7da2611fdc1e39301da02bbc68e62c93 WatchSource:0}: Error finding container fcff4cfd1863d673f74002561cf0d11d7da2611fdc1e39301da02bbc68e62c93: Status 404 returned error can't find the container with id fcff4cfd1863d673f74002561cf0d11d7da2611fdc1e39301da02bbc68e62c93
Sep 14 11:00:55.004045 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.003881    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fe2c4f_cf79_4633_8472_41a9f27c41fe.slice/crio-b699e3499cb4f3076b51ccd203f75d7c63cb75b3bb0ae17804c07f6ffe23cba6 WatchSource:0}: Error finding container b699e3499cb4f3076b51ccd203f75d7c63cb75b3bb0ae17804c07f6ffe23cba6: Status 404 returned error can't find the container with id b699e3499cb4f3076b51ccd203f75d7c63cb75b3bb0ae17804c07f6ffe23cba6
Sep 14 11:00:55.004810 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.004789    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aabd4d_bdfe_451b_a3e3_c5252baf74b8.slice/crio-ab3b044725e30e817aef0000320619536aa28efb073b29daa2b8de4160f40706 WatchSource:0}: Error finding container ab3b044725e30e817aef0000320619536aa28efb073b29daa2b8de4160f40706: Status 404 returned error can't find the container with id ab3b044725e30e817aef0000320619536aa28efb073b29daa2b8de4160f40706
Sep 14 11:00:55.006492 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.006331    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b88b4a5_4f6f_46c6_aa3a_09f2b3cd89ea.slice/crio-21453fcab8d143a50c8a2a8ec1ae911673368bf4dca88f78baa0ce60ade916d6 WatchSource:0}: Error finding container 21453fcab8d143a50c8a2a8ec1ae911673368bf4dca88f78baa0ce60ade916d6: Status 404 returned error can't find the container with id 21453fcab8d143a50c8a2a8ec1ae911673368bf4dca88f78baa0ce60ade916d6
Sep 14 11:00:55.007219 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.007182    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda162ae_636e_43a3_ab3e_5dafb5351179.slice/crio-094f2227aa083a8b9d616efb6fd5ff99be2b63531bc14eb3cfdb44c8b8b55704 WatchSource:0}: Error finding container 094f2227aa083a8b9d616efb6fd5ff99be2b63531bc14eb3cfdb44c8b8b55704: Status 404 returned error can't find the container with id 094f2227aa083a8b9d616efb6fd5ff99be2b63531bc14eb3cfdb44c8b8b55704
Sep 14 11:00:55.009790 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:00:55.008595    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6d424e_e203_4fae_9ff5_db0982a2f37b.slice/crio-820a214c8da8bdb982d5ec118faa0f2160819d4252511d5d58b56d1b86423caf WatchSource:0}: Error finding container 820a214c8da8bdb982d5ec118faa0f2160819d4252511d5d58b56d1b86423caf: Status 404 returned error can't find the container with id 820a214c8da8bdb982d5ec118faa0f2160819d4252511d5d58b56d1b86423caf
Sep 14 11:00:55.412652 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.412619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:55.412782 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.412733    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:55.412825 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.412795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:00:57.412777013 +0000 UTC m=+6.035330210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:55.513877 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.513790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:55.513877 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.513858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:55.514087 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.513978    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:55.514087 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.514044    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:57.514025824 +0000 UTC m=+6.136579036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:55.514087 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.513977    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:00:55.514087 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.514080    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:00:55.514290 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.514093    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:55.514290 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.514142    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:00:57.514130002 +0000 UTC m=+6.136683193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:55.840121 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.840031    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:55:53 +0000 UTC" deadline="2028-05-31 04:07:12.859714016 +0000 UTC"
Sep 14 11:00:55.840121 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.840075    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14993h6m17.019643006s"
Sep 14 11:00:55.924553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.924521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:55.924722 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.924684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:00:55.925182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.925159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:55.925298 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.925278    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:00:55.925404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.925373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:55.927788 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:55.927760    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:00:55.932602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.932532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerStarted","Data":"fcff4cfd1863d673f74002561cf0d11d7da2611fdc1e39301da02bbc68e62c93"}
Sep 14 11:00:55.934375 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.934352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lglg" event={"ID":"422e099d-bace-421a-a944-8f29a52a5ff5","Type":"ContainerStarted","Data":"de2c7399cfbe8905b518fac199a85a3e41d111cce95b46240d7bf663c5c7eaf8"}
Sep 14 11:00:55.938727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.938703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"21453fcab8d143a50c8a2a8ec1ae911673368bf4dca88f78baa0ce60ade916d6"}
Sep 14 11:00:55.940483 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.940460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8" event={"ID":"0983107e-95db-4d04-9e20-d4e4f17a3139","Type":"ContainerStarted","Data":"29ec53efc4e64becdc054f766ba3b90f122e13dce703352b66664ee3766a2700"}
Sep 14 11:00:55.944965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.944943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d8dg5" event={"ID":"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41","Type":"ContainerStarted","Data":"26f0638412b9012ade105144e863eae71f65ac8aaf5a95efbd82877383093a6f"}
Sep 14 11:00:55.951683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.951659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal" event={"ID":"ed43c03ec92d5b25a955c89d154a23b6","Type":"ContainerStarted","Data":"780a3707386b0d58dc7fe43a3a3c61003c04834efdc8a7b6cb65e0360eca47eb"}
Sep 14 11:00:55.956985 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.956962    2563 generic.go:356] "Generic (PLEG): container finished" podID="45284353ad5e38e301b33a8146d7b581" containerID="83319418725d0646e2b1cb36e6c79da5e6ab79330651fac09fb8e866e2efb0cf" exitCode=0
Sep 14 11:00:55.957077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.957041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal" event={"ID":"45284353ad5e38e301b33a8146d7b581","Type":"ContainerDied","Data":"83319418725d0646e2b1cb36e6c79da5e6ab79330651fac09fb8e866e2efb0cf"}
Sep 14 11:00:55.962276 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.962232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7psjm" event={"ID":"0a6d424e-e203-4fae-9ff5-db0982a2f37b","Type":"ContainerStarted","Data":"820a214c8da8bdb982d5ec118faa0f2160819d4252511d5d58b56d1b86423caf"}
Sep 14 11:00:55.967300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.967275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sr4fm" event={"ID":"cda162ae-636e-43a3-ab3e-5dafb5351179","Type":"ContainerStarted","Data":"094f2227aa083a8b9d616efb6fd5ff99be2b63531bc14eb3cfdb44c8b8b55704"}
Sep 14 11:00:55.970721 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.970687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbccv" event={"ID":"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8","Type":"ContainerStarted","Data":"ab3b044725e30e817aef0000320619536aa28efb073b29daa2b8de4160f40706"}
Sep 14 11:00:55.973456 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:55.973434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vcv6v" event={"ID":"98fe2c4f-cf79-4633-8472-41a9f27c41fe","Type":"ContainerStarted","Data":"b699e3499cb4f3076b51ccd203f75d7c63cb75b3bb0ae17804c07f6ffe23cba6"}
Sep 14 11:00:56.012328 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:56.012276    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-154.ec2.internal" podStartSLOduration=3.012257933 podStartE2EDuration="3.012257933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:00:55.976663276 +0000 UTC m=+4.599216491" watchObservedRunningTime="2026-09-14 11:00:56.012257933 +0000 UTC m=+4.634811148"
Sep 14 11:00:56.982363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:56.982325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal" event={"ID":"45284353ad5e38e301b33a8146d7b581","Type":"ContainerStarted","Data":"0ab96677867dd7fc1c21c311082862d2fc4a62def4edb2d02f91ad3cc1f9fcf1"}
Sep 14 11:00:57.027886 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.027792    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-154.ec2.internal" podStartSLOduration=5.027771689 podStartE2EDuration="5.027771689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:00:57.026534668 +0000 UTC m=+5.649087884" watchObservedRunningTime="2026-09-14 11:00:57.027771689 +0000 UTC m=+5.650324907"
Sep 14 11:00:57.438085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.438003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:57.438241 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.438147    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:57.438241 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.438213    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:01.438193733 +0000 UTC m=+10.060746931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:00:57.538670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.538633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:57.538853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.538726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:57.538970 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.538871    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:00:57.538970 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.538893    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:00:57.538970 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.538905    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:57.538970 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.538959    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:01:01.538941446 +0000 UTC m=+10.161494648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:00:57.539366 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.539344    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:57.539467 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.539418    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:01.539402603 +0000 UTC m=+10.161955819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:00:57.920799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.920224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:57.920799 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.920345    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:00:57.920799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.920599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:57.920799 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.920685    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:00:57.920799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:57.920705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:57.920799 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:57.920786    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:00:59.913553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:59.913060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:00:59.913553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:59.913071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:00:59.913553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:00:59.913193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:00:59.913553 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:59.913195    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:00:59.913553 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:59.913301    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:00:59.913553 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:00:59.913382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:01.472546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:01.471735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:01.472546 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.471831    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:01:01.472546 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.472177    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:09.472153564 +0000 UTC m=+18.094706816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:01:01.573245 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:01.573210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:01.573437 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:01.573264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:01.573437 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.573351    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:01:01.573437 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.573380    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:01:01.573437 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.573417    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:01:01.573437 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.573421    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:09.573406752 +0000 UTC m=+18.195959963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:01:01.573437 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.573427    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:01:01.573716 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.573472    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:01:09.573456429 +0000 UTC m=+18.196009634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:01:01.915686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:01.915597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:01.915686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:01.915658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:01.915895 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.915767    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:01.915895 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:01.915784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:01.915895 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.915881    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:01.916048 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:01.915973    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:03.912849 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:03.912804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:03.913233 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:03.912804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:03.913233 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:03.912928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:03.913233 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:03.912804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:03.913233 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:03.913006    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:03.913233 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:03.913071    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:05.912905 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:05.912622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:05.912905 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:05.912623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:05.912905 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:05.912740    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:05.912905 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:05.912803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:05.912905 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:05.912904    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:05.913636 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:05.912965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:07.912344 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:07.912309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:07.912823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:07.912353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:07.912823 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:07.912459    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:07.912823 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:07.912542    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:07.912823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:07.912599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:07.912823 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:07.912689    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:09.532488 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:09.532456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:09.532938 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.532563    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:01:09.532938 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.532633    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:25.53261317 +0000 UTC m=+34.155166367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:01:09.633300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:09.633266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:09.633473 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:09.633339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:09.633473 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.633454    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:01:09.633608 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.633480    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:01:09.633608 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.633498    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:01:09.633608 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.633510    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:01:09.633608 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.633526    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:25.633505461 +0000 UTC m=+34.256058657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:01:09.633608 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.633556    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:01:25.63354183 +0000 UTC m=+34.256095035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:01:09.912575 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:09.912485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:09.912726 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:09.912497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:09.912726 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.912613    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:09.912726 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.912711    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:09.912873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:09.912501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:09.912873 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:09.912780    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:11.913020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:11.912981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:11.913496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:11.913102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:11.913496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:11.913121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:11.913496 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:11.913097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:11.913496 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:11.913227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:11.913496 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:11.913325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:13.912990 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:13.912790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:13.913874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:13.912790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:13.913874 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:13.913061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:13.913874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:13.912828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:13.913874 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:13.913198    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:13.913874 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:13.913329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:14.021216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.021187    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9afc335-7d75-48cf-8e86-1e41ef2df1fa" containerID="7c217f02fbb8fd9db394820833f18d93e0af27b022fa04a18ced611caca569a6" exitCode=0
Sep 14 11:01:14.021318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.021263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerDied","Data":"7c217f02fbb8fd9db394820833f18d93e0af27b022fa04a18ced611caca569a6"}
Sep 14 11:01:14.022852 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.022829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lglg" event={"ID":"422e099d-bace-421a-a944-8f29a52a5ff5","Type":"ContainerStarted","Data":"829f5022cfba6315447cf7346f90f6e490a56ca204514fcdfe01e49ff30ca886"}
Sep 14 11:01:14.026164 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.026143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"7e795854a002598c04729f36866401e992dd702ec460f2224e964dad0fa91ec4"}
Sep 14 11:01:14.026242 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.026202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"7840bee6ed36f007a92d006cb790dd5a96ad50f270f863e8decc912ba6809631"}
Sep 14 11:01:14.026292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.026248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"cdc46a8c28e2a57e52b9a88eb90fbd323e3ddf87a0968947598275c98f77f289"}
Sep 14 11:01:14.026292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.026281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"3aec9d2ee5941c670b9910659f27477b57144fa1e17d23159848ca40d0d7ec1a"}
Sep 14 11:01:14.026380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.026309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"740a8c217f1fdd83d938456180576cf565b25951c1d9ff8f5a988df5054ce9ec"}
Sep 14 11:01:14.026380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.026336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"01d3b9d4dfe24351018f2ab257138536eb6b8c60c69ce4e7574ee0f68cdc2faa"}
Sep 14 11:01:14.027958 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.027733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8" event={"ID":"0983107e-95db-4d04-9e20-d4e4f17a3139","Type":"ContainerStarted","Data":"e99863b13eb7b0eba3ed6bf01aaeef7b90e6d2ca4b6f8b001e58bb979b8b5600"}
Sep 14 11:01:14.029271 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.029247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d8dg5" event={"ID":"2c60ef6d-94b5-4a01-afb1-99c1f6e99d41","Type":"ContainerStarted","Data":"18b54e602baeedd675553151a8364d3877333c2f3ac22d37194b10d46bebdf32"}
Sep 14 11:01:14.030826 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.030798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7psjm" event={"ID":"0a6d424e-e203-4fae-9ff5-db0982a2f37b","Type":"ContainerStarted","Data":"57fdfcbace36441ab0fa1e96ea094a89498efb07500d7edf3202330789406bb6"}
Sep 14 11:01:14.032540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.032499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbccv" event={"ID":"f8aabd4d-bdfe-451b-a3e3-c5252baf74b8","Type":"ContainerStarted","Data":"f7ad9dff1f273096d792220aa5e6449fcfe2feadc8443a55bef6d9a6bf432ace"}
Sep 14 11:01:14.034357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.034313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vcv6v" event={"ID":"98fe2c4f-cf79-4633-8472-41a9f27c41fe","Type":"ContainerStarted","Data":"f14a4d522bf1177c7d0b635b482a2751e45eb04954ae36548031778661bbf473"}
Sep 14 11:01:14.074831 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.074790    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vcv6v" podStartSLOduration=4.16981939 podStartE2EDuration="22.074777598s" totalImagesPullingTime="17.904958208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:14.074424915 +0000 UTC m=+22.696978129" watchObservedRunningTime="2026-09-14 11:01:14.074777598 +0000 UTC m=+22.697330812"
Sep 14 11:01:14.093800 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.093764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d8dg5" podStartSLOduration=4.107507563 podStartE2EDuration="22.09375328s" totalImagesPullingTime="17.986245717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:14.093226902 +0000 UTC m=+22.715780116" watchObservedRunningTime="2026-09-14 11:01:14.09375328 +0000 UTC m=+22.716306494"
Sep 14 11:01:14.110312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.110279    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7psjm" podStartSLOduration=4.197464957 podStartE2EDuration="22.110265756s" totalImagesPullingTime="17.912800799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:14.10970115 +0000 UTC m=+22.732254364" watchObservedRunningTime="2026-09-14 11:01:14.110265756 +0000 UTC m=+22.732818970"
Sep 14 11:01:14.112775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.112756    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 11:01:14.137646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.137610    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jbccv" podStartSLOduration=4.132019532 podStartE2EDuration="22.137597704s" totalImagesPullingTime="18.005578172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:14.137033359 +0000 UTC m=+22.759586575" watchObservedRunningTime="2026-09-14 11:01:14.137597704 +0000 UTC m=+22.760150896"
Sep 14 11:01:14.154153 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.154120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4lglg" podStartSLOduration=4.246209113 podStartE2EDuration="22.154111716s" totalImagesPullingTime="17.907902603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:14.153420707 +0000 UTC m=+22.775973924" watchObservedRunningTime="2026-09-14 11:01:14.154111716 +0000 UTC m=+22.776664907"
Sep 14 11:01:14.869578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.869494    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T11:01:14.112769181Z","UUID":"28c375db-b9e0-4e07-bea9-96fba180b597","Handler":null,"Name":"","Endpoint":""}
Sep 14 11:01:14.871354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.871334    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 11:01:14.871485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.871367    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 11:01:14.972767 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.972732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:01:14.973423 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:14.973383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:01:15.039766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.039730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8" event={"ID":"0983107e-95db-4d04-9e20-d4e4f17a3139","Type":"ContainerStarted","Data":"00f476d79faa47c7e22fe6b15f7ec07fd7477925a7a6666b7857e69e7215ba7f"}
Sep 14 11:01:15.039933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.039800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8" event={"ID":"0983107e-95db-4d04-9e20-d4e4f17a3139","Type":"ContainerStarted","Data":"473ba40a52cc381d76279441320e5be2c9a02cb373ca39223daeb5385d4888c6"}
Sep 14 11:01:15.041421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.041374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sr4fm" event={"ID":"cda162ae-636e-43a3-ab3e-5dafb5351179","Type":"ContainerStarted","Data":"af955f26e175265a3656d4d1a89d7d7b7a18eecd680ffa0f50ff27c1455e9ef6"}
Sep 14 11:01:15.042411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.042372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:01:15.042524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.042460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4lglg"
Sep 14 11:01:15.063693 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.063631    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzb8" podStartSLOduration=3.532163995 podStartE2EDuration="23.063613425s" totalImagesPullingTime="19.53144943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:15.062668672 +0000 UTC m=+23.685221882" watchObservedRunningTime="2026-09-14 11:01:15.063613425 +0000 UTC m=+23.686166643"
Sep 14 11:01:15.082248 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.082163    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sr4fm" podStartSLOduration=6.16726148 podStartE2EDuration="24.082147101s" totalImagesPullingTime="17.914885621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:15.081300149 +0000 UTC m=+23.703853362" watchObservedRunningTime="2026-09-14 11:01:15.082147101 +0000 UTC m=+23.704700315"
Sep 14 11:01:15.912540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.912452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:15.912540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.912495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:15.912540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:15.912463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:15.912810 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:15.912574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:15.912810 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:15.912722    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:15.912909 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:15.912828    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:16.048156 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:16.048119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"f50be7566f4a8d48a767bb309dda9782b857c5cf87c7ca9f72f24e5dbf222481"}
Sep 14 11:01:16.558571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:16.558535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:17.912690 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:17.912604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:17.912690 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:17.912640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:17.913177 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:17.912611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:17.913177 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:17.912735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:17.913177 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:17.912816    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:17.913177 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:17.912934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:19.059711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.058884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" event={"ID":"5b88b4a5-4f6f-46c6-aa3a-09f2b3cd89ea","Type":"ContainerStarted","Data":"054225886b075be4b1f117458ee70e72eee5516a097718e3d963be8b6286909f"}
Sep 14 11:01:19.059711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.059274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:01:19.059711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.059336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:01:19.059711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.059384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:01:19.061105 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.061080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerStarted","Data":"82d1a0b40bf64cea743b87c753c987206004b17cd0fd3a825f401d4cd298b788"}
Sep 14 11:01:19.074518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.074491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:01:19.074653 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.074626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:01:19.121699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.121656    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz" podStartSLOduration=10.15818077 podStartE2EDuration="28.121643492s" totalImagesPullingTime="17.963462722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:19.094695708 +0000 UTC m=+27.717248922" watchObservedRunningTime="2026-09-14 11:01:19.121643492 +0000 UTC m=+27.744196713"
Sep 14 11:01:19.862663 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.862632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:19.912699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.912670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:19.912859 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.912670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:19.912859 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:19.912768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:19.912859 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:19.912774    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:19.912859 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:19.912846    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:19.913000 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:19.912915    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:20.065648 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.065613    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9afc335-7d75-48cf-8e86-1e41ef2df1fa" containerID="82d1a0b40bf64cea743b87c753c987206004b17cd0fd3a825f401d4cd298b788" exitCode=0
Sep 14 11:01:20.066098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.065693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerDied","Data":"82d1a0b40bf64cea743b87c753c987206004b17cd0fd3a825f401d4cd298b788"}
Sep 14 11:01:20.796305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.796058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7sw9"]
Sep 14 11:01:20.796533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.796458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:20.796610 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:20.796558    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:20.796932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.796908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jplcp"]
Sep 14 11:01:20.797057 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.797011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:20.797126 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:20.797099    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:20.797576 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.797555    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jrkpt"]
Sep 14 11:01:20.797680 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:20.797655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:20.797759 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:20.797738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:21.770608 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:21.770576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:21.844330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:21.844302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:21.913656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:21.913621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:21.913823 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:21.913735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:22.014238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:22.014175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:22.115633 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:22.115608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:22.913116 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:22.913019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:22.913116 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:22.913018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:22.913716 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:22.913147    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:22.913716 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:22.913206    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:23.075433 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:23.075379    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9afc335-7d75-48cf-8e86-1e41ef2df1fa" containerID="810ce524e020695c6aa01686d4a64c9bbe402e2aabc5655b2b606a706a0b3947" exitCode=0
Sep 14 11:01:23.075600 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:23.075464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerDied","Data":"810ce524e020695c6aa01686d4a64c9bbe402e2aabc5655b2b606a706a0b3947"}
Sep 14 11:01:23.912621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:23.912531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:23.912783 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:23.912629    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7sw9" podUID="82f1808b-0670-4a82-8c87-39180882715e"
Sep 14 11:01:24.912659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:24.912626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:24.913069 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:24.912746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrkpt" podUID="78e7293f-aa46-4991-910b-60558d5098bb"
Sep 14 11:01:24.913069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:24.912795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:24.913069 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:24.912895    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jplcp" podUID="067aa408-be71-402b-aada-cd64b57c95f0"
Sep 14 11:01:25.224595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.224519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:25.291563 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.291534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:25.556161 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.556132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:25.556314 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.556294    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 11:01:25.556382 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.556375    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret podName:82f1808b-0670-4a82-8c87-39180882715e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:57.556351995 +0000 UTC m=+66.178905208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret") pod "global-pull-secret-syncer-x7sw9" (UID: "82f1808b-0670-4a82-8c87-39180882715e") : object "kube-system"/"original-pull-secret" not registered
Sep 14 11:01:25.657483 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.657458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:25.657589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.657526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:25.657651 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.657617    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 11:01:25.657651 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.657641    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 11:01:25.657742 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.657650    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:01:25.657742 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.657716    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:57.657695914 +0000 UTC m=+66.280249121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 11:01:25.657742 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.657653    2563 projected.go:196] Error preparing data for projected volume kube-api-access-vf64c for pod openshift-network-diagnostics/network-check-target-jrkpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:01:25.657890 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.657782    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c podName:78e7293f-aa46-4991-910b-60558d5098bb nodeName:}" failed. No retries permitted until 2026-09-14 11:01:57.657767344 +0000 UTC m=+66.280320552 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vf64c" (UniqueName: "kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c") pod "network-check-target-jrkpt" (UID: "78e7293f-aa46-4991-910b-60558d5098bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 11:01:25.671643 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.671625    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-154.ec2.internal" event="NodeReady"
Sep 14 11:01:25.671733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.671724    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 11:01:25.740870 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.740808    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-745q5"]
Sep 14 11:01:25.765494 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.765467    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9lhw2"]
Sep 14 11:01:25.765631 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.765588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.767627 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.767608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpbn2\""
Sep 14 11:01:25.767925 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.767908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 11:01:25.768015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.767997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 11:01:25.780211 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.780193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-745q5"]
Sep 14 11:01:25.780277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.780225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9lhw2"]
Sep 14 11:01:25.780318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.780291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:25.784586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.784566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 11:01:25.784692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.784676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 11:01:25.784873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.784858    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 11:01:25.784942 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.784922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qzhb6\""
Sep 14 11:01:25.859572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.859535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqrm\" (UniqueName: \"kubernetes.io/projected/449a4fc3-67e5-41ad-b69f-33551d650e3a-kube-api-access-5jqrm\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.859726 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.859579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a4fc3-67e5-41ad-b69f-33551d650e3a-tmp-dir\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.859726 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.859616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.859726 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.859665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/449a4fc3-67e5-41ad-b69f-33551d650e3a-config-volume\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.862403 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.862368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:25.912689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.912665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:25.916904 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.916880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 11:01:25.959932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.959911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/449a4fc3-67e5-41ad-b69f-33551d650e3a-config-volume\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.960039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.959957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqrm\" (UniqueName: \"kubernetes.io/projected/449a4fc3-67e5-41ad-b69f-33551d650e3a-kube-api-access-5jqrm\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.960090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a4fc3-67e5-41ad-b69f-33551d650e3a-tmp-dir\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.960090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02c8187f-5f7e-4860-a58f-65496fd89b31-cert\") pod \"ingress-canary-9lhw2\" (UID: \"02c8187f-5f7e-4860-a58f-65496fd89b31\") " pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:25.960184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.960184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl4f\" (UniqueName: \"kubernetes.io/projected/02c8187f-5f7e-4860-a58f-65496fd89b31-kube-api-access-wnl4f\") pod \"ingress-canary-9lhw2\" (UID: \"02c8187f-5f7e-4860-a58f-65496fd89b31\") " pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:25.960274 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.960237    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:25.960326 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:25.960305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:26.460286115 +0000 UTC m=+35.082839325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:25.960429 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:25.960496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a4fc3-67e5-41ad-b69f-33551d650e3a-tmp-dir\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.960675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.960656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/449a4fc3-67e5-41ad-b69f-33551d650e3a-config-volume\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:25.971441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:25.971416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqrm\" (UniqueName: \"kubernetes.io/projected/449a4fc3-67e5-41ad-b69f-33551d650e3a-kube-api-access-5jqrm\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:26.060595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.060536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl4f\" (UniqueName: \"kubernetes.io/projected/02c8187f-5f7e-4860-a58f-65496fd89b31-kube-api-access-wnl4f\") pod \"ingress-canary-9lhw2\" (UID: \"02c8187f-5f7e-4860-a58f-65496fd89b31\") " pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:26.060595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.060589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02c8187f-5f7e-4860-a58f-65496fd89b31-cert\") pod \"ingress-canary-9lhw2\" (UID: \"02c8187f-5f7e-4860-a58f-65496fd89b31\") " pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:26.062859 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.062839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02c8187f-5f7e-4860-a58f-65496fd89b31-cert\") pod \"ingress-canary-9lhw2\" (UID: \"02c8187f-5f7e-4860-a58f-65496fd89b31\") " pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:26.072936 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.072915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl4f\" (UniqueName: \"kubernetes.io/projected/02c8187f-5f7e-4860-a58f-65496fd89b31-kube-api-access-wnl4f\") pod \"ingress-canary-9lhw2\" (UID: \"02c8187f-5f7e-4860-a58f-65496fd89b31\") " pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:26.083857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.083832    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9afc335-7d75-48cf-8e86-1e41ef2df1fa" containerID="b85c81f8547c5e7550eaf2ef26b3e7383e972dcc4832d6d1d8af51559aed3aac" exitCode=0
Sep 14 11:01:26.083948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.083886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerDied","Data":"b85c81f8547c5e7550eaf2ef26b3e7383e972dcc4832d6d1d8af51559aed3aac"}
Sep 14 11:01:26.087270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.087256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9lhw2"
Sep 14 11:01:26.405278 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.405200    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9lhw2"]
Sep 14 11:01:26.408185 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:01:26.408157    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c8187f_5f7e_4860_a58f_65496fd89b31.slice/crio-1176b1a9a392c494072a1b7336dcec47c9f0401f5c693be1be2810cbac6386a6 WatchSource:0}: Error finding container 1176b1a9a392c494072a1b7336dcec47c9f0401f5c693be1be2810cbac6386a6: Status 404 returned error can't find the container with id 1176b1a9a392c494072a1b7336dcec47c9f0401f5c693be1be2810cbac6386a6
Sep 14 11:01:26.463432 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.463379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:26.463556 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:26.463514    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:26.463607 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:26.463581    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:27.463567689 +0000 UTC m=+36.086120881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:26.912781 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.912755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:26.913546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.912793    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:26.915346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.915326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 11:01:26.915978 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.915949    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 11:01:26.916070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.915962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 11:01:26.916070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.915987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mckdx\""
Sep 14 11:01:26.916070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:26.916004    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npv4p\""
Sep 14 11:01:27.088323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:27.088279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9lhw2" event={"ID":"02c8187f-5f7e-4860-a58f-65496fd89b31","Type":"ContainerStarted","Data":"1176b1a9a392c494072a1b7336dcec47c9f0401f5c693be1be2810cbac6386a6"}
Sep 14 11:01:27.469950 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:27.469914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:27.470130 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:27.470086    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:27.470179 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:27.470165    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:29.470144196 +0000 UTC m=+38.092697392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:29.483988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:29.483948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:29.484468 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:29.484134    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:29.484468 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:29.484452    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:33.484430645 +0000 UTC m=+42.106983841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:30.905124 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:30.905079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:31.007573 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:31.007528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:31.536963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:31.536932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:31.638009 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:31.637975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:32.102816 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:32.102784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9lhw2" event={"ID":"02c8187f-5f7e-4860-a58f-65496fd89b31","Type":"ContainerStarted","Data":"68333f7c0c039f038d3995696dfdfbb280ae5377cbb1633688d324f602077aa6"}
Sep 14 11:01:32.105727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:32.105702    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9afc335-7d75-48cf-8e86-1e41ef2df1fa" containerID="1d8cdb305450837f3f74bbb3cff592e121ee0a73a1bd9118ec6c7543b9d127fc" exitCode=0
Sep 14 11:01:32.105843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:32.105754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerDied","Data":"1d8cdb305450837f3f74bbb3cff592e121ee0a73a1bd9118ec6c7543b9d127fc"}
Sep 14 11:01:32.118859 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:32.118811    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9lhw2" podStartSLOduration=2.027812094 podStartE2EDuration="7.118794267s" totalImagesPullingTime="5.090982173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:32.118483952 +0000 UTC m=+40.741037169" watchObservedRunningTime="2026-09-14 11:01:32.118794267 +0000 UTC m=+40.741347481"
Sep 14 11:01:33.111199 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:33.111167    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9afc335-7d75-48cf-8e86-1e41ef2df1fa" containerID="b67882d7764f52e30e9dd5701e7502bcfc77c755b4660cea90e4c87606b3a2d2" exitCode=0
Sep 14 11:01:33.111560 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:33.111240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerDied","Data":"b67882d7764f52e30e9dd5701e7502bcfc77c755b4660cea90e4c87606b3a2d2"}
Sep 14 11:01:33.512754 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:33.512654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:33.512925 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:33.512786    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:33.512925 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:33.512854    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:41.512836417 +0000 UTC m=+50.135389623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:34.116267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:34.116234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" event={"ID":"c9afc335-7d75-48cf-8e86-1e41ef2df1fa","Type":"ContainerStarted","Data":"17c67d7b1764c21b6e2820ee97408c2053573642c023b196a6c74f3ab62338c0"}
Sep 14 11:01:34.141280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:34.141239    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fqzz9" podStartSLOduration=9.631149038 podStartE2EDuration="42.141224782s" totalImagesPullingTime="32.510075744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:34.140434209 +0000 UTC m=+42.762987426" watchObservedRunningTime="2026-09-14 11:01:34.141224782 +0000 UTC m=+42.763777993"
Sep 14 11:01:34.266399 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:34.266359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:34.337366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:34.337341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:35.034431 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:35.034356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:35.134445 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:35.134402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:38.270792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:38.270760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:38.335943 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:38.335909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:38.400497 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:38.400469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:41.563702 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:41.563656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:41.564074 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:41.563776    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:41.564074 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:41.563867    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:57.563849513 +0000 UTC m=+66.186402719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:44.689927 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:44.689892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:44.748604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:44.748575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:51.078890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:51.078853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h28cz"
Sep 14 11:01:51.236272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:51.236234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:51.298788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:51.298754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:57.182167 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.182134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:57.283030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.282998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:57.386404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.386372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:57.581562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.581456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:57.581562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.581522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:01:57.581818 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:57.581627    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 11:01:57.581818 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:57.581675    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls podName:449a4fc3-67e5-41ad-b69f-33551d650e3a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:29.581661527 +0000 UTC m=+98.204214719 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls") pod "dns-default-745q5" (UID: "449a4fc3-67e5-41ad-b69f-33551d650e3a") : secret "dns-default-metrics-tls" not found
Sep 14 11:01:57.583916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.583895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 11:01:57.595271 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.595243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82f1808b-0670-4a82-8c87-39180882715e-original-pull-secret\") pod \"global-pull-secret-syncer-x7sw9\" (UID: \"82f1808b-0670-4a82-8c87-39180882715e\") " pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:57.682609 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.682567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:57.682779 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.682626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:01:57.685454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.685439    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 11:01:57.685503 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.685482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 11:01:57.693278 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:57.693247    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 14 11:01:57.693339 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:57.693313    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs podName:067aa408-be71-402b-aada-cd64b57c95f0 nodeName:}" failed. No retries permitted until 2026-09-14 11:03:01.693297321 +0000 UTC m=+130.315850512 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs") pod "network-metrics-daemon-jplcp" (UID: "067aa408-be71-402b-aada-cd64b57c95f0") : secret "metrics-daemon-secret" not found
Sep 14 11:01:57.695454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.695439    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 11:01:57.706044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.706023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf64c\" (UniqueName: \"kubernetes.io/projected/78e7293f-aa46-4991-910b-60558d5098bb-kube-api-access-vf64c\") pod \"network-check-target-jrkpt\" (UID: \"78e7293f-aa46-4991-910b-60558d5098bb\") " pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:57.724163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.724138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7sw9"
Sep 14 11:01:57.830439 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.829985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mckdx\""
Sep 14 11:01:57.837423 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.837334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:01:57.995404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:57.995361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7sw9"]
Sep 14 11:01:57.998227 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:01:57.998197    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f1808b_0670_4a82_8c87_39180882715e.slice/crio-3f7a274f353c639159577cd7f93ff6e123ef5ebc35e29075c1672ce94120feb4 WatchSource:0}: Error finding container 3f7a274f353c639159577cd7f93ff6e123ef5ebc35e29075c1672ce94120feb4: Status 404 returned error can't find the container with id 3f7a274f353c639159577cd7f93ff6e123ef5ebc35e29075c1672ce94120feb4
Sep 14 11:01:58.105355 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.105325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jrkpt"]
Sep 14 11:01:58.108622 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:01:58.108594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e7293f_aa46_4991_910b_60558d5098bb.slice/crio-cb171575ebd86cdebb5b27d62edfad54897ce2d1f6eec54fff785f0a446c80f0 WatchSource:0}: Error finding container cb171575ebd86cdebb5b27d62edfad54897ce2d1f6eec54fff785f0a446c80f0: Status 404 returned error can't find the container with id cb171575ebd86cdebb5b27d62edfad54897ce2d1f6eec54fff785f0a446c80f0
Sep 14 11:01:58.170614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.170579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jrkpt" event={"ID":"78e7293f-aa46-4991-910b-60558d5098bb","Type":"ContainerStarted","Data":"cb171575ebd86cdebb5b27d62edfad54897ce2d1f6eec54fff785f0a446c80f0"}
Sep 14 11:01:58.171665 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.171640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7sw9" event={"ID":"82f1808b-0670-4a82-8c87-39180882715e","Type":"ContainerStarted","Data":"3f7a274f353c639159577cd7f93ff6e123ef5ebc35e29075c1672ce94120feb4"}
Sep 14 11:01:58.801659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.801623    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"]
Sep 14 11:01:58.804914 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.804896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"
Sep 14 11:01:58.808075 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.808052    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:01:58.808190 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.808052    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m9rwr\""
Sep 14 11:01:58.808190 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.808151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 14 11:01:58.816332 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.816310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"]
Sep 14 11:01:58.890848 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.890810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dn2\" (UniqueName: \"kubernetes.io/projected/f1d017a4-92d9-4e85-9d7e-bb8f64c2265a-kube-api-access-64dn2\") pod \"volume-data-source-validator-86695b7cc7-6gwgh\" (UID: \"f1d017a4-92d9-4e85-9d7e-bb8f64c2265a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"
Sep 14 11:01:58.913038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.913003    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-hnnpq"]
Sep 14 11:01:58.916250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.916226    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"]
Sep 14 11:01:58.916380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.916363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:58.919487 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.919462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:58.927485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.927462    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 14 11:01:58.927613 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.927463    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 11:01:58.927687 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.927621    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 14 11:01:58.927767 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.927463    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 11:01:58.928291 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.927947    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:01:58.928291 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.928179    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bmxb5\""
Sep 14 11:01:58.928793 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.928726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 14 11:01:58.929077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.929053    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"]
Sep 14 11:01:58.929227 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.929192    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xqwrb\""
Sep 14 11:01:58.930091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.930072    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 14 11:01:58.932186 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.932167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:58.942240 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.942219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 14 11:01:58.942413 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.942365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 14 11:01:58.943678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.943054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 14 11:01:58.943678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.943274    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 11:01:58.943678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.943528    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rfxk6\""
Sep 14 11:01:58.943678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.943646    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 11:01:58.943928 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.943876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"]
Sep 14 11:01:58.944673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.944656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hnnpq"]
Sep 14 11:01:58.987190 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.987121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"]
Sep 14 11:01:58.992038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8fr\" (UniqueName: \"kubernetes.io/projected/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-kube-api-access-4g8fr\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:58.992194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:58.992194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dn2\" (UniqueName: \"kubernetes.io/projected/f1d017a4-92d9-4e85-9d7e-bb8f64c2265a-kube-api-access-64dn2\") pod \"volume-data-source-validator-86695b7cc7-6gwgh\" (UID: \"f1d017a4-92d9-4e85-9d7e-bb8f64c2265a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"
Sep 14 11:01:58.992194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbefa077-d861-42b7-be43-c4c3b0bdfaba-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:58.992194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbefa077-d861-42b7-be43-c4c3b0bdfaba-service-ca-bundle\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:58.992194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbefa077-d861-42b7-be43-c4c3b0bdfaba-serving-cert\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:58.992484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkw7\" (UniqueName: \"kubernetes.io/projected/9ce0c132-d207-4e7a-bad5-d129defd0572-kube-api-access-5jkw7\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:58.992484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbefa077-d861-42b7-be43-c4c3b0bdfaba-tmp\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:58.992484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cfz\" (UniqueName: \"kubernetes.io/projected/cbefa077-d861-42b7-be43-c4c3b0bdfaba-kube-api-access-z4cfz\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:58.992484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:58.992484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:58.992484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:58.992473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbefa077-d861-42b7-be43-c4c3b0bdfaba-snapshots\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.006082 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.006050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dn2\" (UniqueName: \"kubernetes.io/projected/f1d017a4-92d9-4e85-9d7e-bb8f64c2265a-kube-api-access-64dn2\") pod \"volume-data-source-validator-86695b7cc7-6gwgh\" (UID: \"f1d017a4-92d9-4e85-9d7e-bb8f64c2265a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"
Sep 14 11:01:59.029934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.029901    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hh9kh"]
Sep 14 11:01:59.033183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.033158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.036651 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.036589    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gvd9b\""
Sep 14 11:01:59.037915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.037851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 14 11:01:59.037915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.037910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:01:59.038086 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.037919    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 14 11:01:59.042007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.041988    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 14 11:01:59.044275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.044252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 14 11:01:59.047602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.047579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hh9kh"]
Sep 14 11:01:59.093774 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:59.093774 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e805c86-98b1-4bc0-b457-b1100590d637-trusted-ca\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.093774 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbefa077-d861-42b7-be43-c4c3b0bdfaba-snapshots\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e805c86-98b1-4bc0-b457-b1100590d637-config\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.094030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8fr\" (UniqueName: \"kubernetes.io/projected/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-kube-api-access-4g8fr\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:59.094030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:59.094030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.093989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zd5\" (UniqueName: \"kubernetes.io/projected/6e805c86-98b1-4bc0-b457-b1100590d637-kube-api-access-f5zd5\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.094030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbefa077-d861-42b7-be43-c4c3b0bdfaba-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbefa077-d861-42b7-be43-c4c3b0bdfaba-service-ca-bundle\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbefa077-d861-42b7-be43-c4c3b0bdfaba-serving-cert\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094319 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.094117    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 11:01:59.094319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkw7\" (UniqueName: \"kubernetes.io/projected/9ce0c132-d207-4e7a-bad5-d129defd0572-kube-api-access-5jkw7\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:59.094319 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.094188    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls podName:9ce0c132-d207-4e7a-bad5-d129defd0572 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:59.594167029 +0000 UTC m=+68.216720234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4v88z" (UID: "9ce0c132-d207-4e7a-bad5-d129defd0572") : secret "samples-operator-tls" not found
Sep 14 11:01:59.094596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbefa077-d861-42b7-be43-c4c3b0bdfaba-tmp\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cfz\" (UniqueName: \"kubernetes.io/projected/cbefa077-d861-42b7-be43-c4c3b0bdfaba-kube-api-access-z4cfz\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:59.094596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cbefa077-d861-42b7-be43-c4c3b0bdfaba-snapshots\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e805c86-98b1-4bc0-b457-b1100590d637-serving-cert\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.094596 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.094561    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 11:01:59.094898 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.094632    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls podName:cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:59.594612759 +0000 UTC m=+68.217165961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gl2m9" (UID: "cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96") : secret "cluster-monitoring-operator-tls" not found
Sep 14 11:01:59.094898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:59.094898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbefa077-d861-42b7-be43-c4c3b0bdfaba-tmp\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.094898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.094865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbefa077-d861-42b7-be43-c4c3b0bdfaba-service-ca-bundle\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.095151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.095097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbefa077-d861-42b7-be43-c4c3b0bdfaba-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.096894 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.096870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbefa077-d861-42b7-be43-c4c3b0bdfaba-serving-cert\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.103662 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.103639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cfz\" (UniqueName: \"kubernetes.io/projected/cbefa077-d861-42b7-be43-c4c3b0bdfaba-kube-api-access-z4cfz\") pod \"insights-operator-5844b856db-hnnpq\" (UID: \"cbefa077-d861-42b7-be43-c4c3b0bdfaba\") " pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.104186 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.104116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8fr\" (UniqueName: \"kubernetes.io/projected/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-kube-api-access-4g8fr\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:59.104484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.104462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkw7\" (UniqueName: \"kubernetes.io/projected/9ce0c132-d207-4e7a-bad5-d129defd0572-kube-api-access-5jkw7\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:59.115153 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.115125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"
Sep 14 11:01:59.175430 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.175298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:59.195029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.194995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zd5\" (UniqueName: \"kubernetes.io/projected/6e805c86-98b1-4bc0-b457-b1100590d637-kube-api-access-f5zd5\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.195250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.195071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e805c86-98b1-4bc0-b457-b1100590d637-serving-cert\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.195250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.195094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e805c86-98b1-4bc0-b457-b1100590d637-trusted-ca\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.195250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.195120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e805c86-98b1-4bc0-b457-b1100590d637-config\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.195912 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.195846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e805c86-98b1-4bc0-b457-b1100590d637-config\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.196261 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.196227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e805c86-98b1-4bc0-b457-b1100590d637-trusted-ca\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.198176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.198151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e805c86-98b1-4bc0-b457-b1100590d637-serving-cert\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.204373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.204345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zd5\" (UniqueName: \"kubernetes.io/projected/6e805c86-98b1-4bc0-b457-b1100590d637-kube-api-access-f5zd5\") pod \"console-operator-696884fb9c-hh9kh\" (UID: \"6e805c86-98b1-4bc0-b457-b1100590d637\") " pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.227064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.227024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hnnpq"
Sep 14 11:01:59.248998 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.248964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:59.349172 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.349082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:01:59.396454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.396351    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh"]
Sep 14 11:01:59.513731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.513701    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hnnpq"]
Sep 14 11:01:59.517158 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:01:59.517125    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbefa077_d861_42b7_be43_c4c3b0bdfaba.slice/crio-96b45f9e0aa2d2f3d9b7430fe3d419ec596ce6e1284200388395c8bf69e9e62a WatchSource:0}: Error finding container 96b45f9e0aa2d2f3d9b7430fe3d419ec596ce6e1284200388395c8bf69e9e62a: Status 404 returned error can't find the container with id 96b45f9e0aa2d2f3d9b7430fe3d419ec596ce6e1284200388395c8bf69e9e62a
Sep 14 11:01:59.598603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.598561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:01:59.598761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.598662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:01:59.598761 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.598688    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 11:01:59.598761 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.598754    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls podName:cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.598736829 +0000 UTC m=+69.221290025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gl2m9" (UID: "cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96") : secret "cluster-monitoring-operator-tls" not found
Sep 14 11:01:59.598941 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.598833    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 11:01:59.598941 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:01:59.598896    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls podName:9ce0c132-d207-4e7a-bad5-d129defd0572 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.598878962 +0000 UTC m=+69.221432178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4v88z" (UID: "9ce0c132-d207-4e7a-bad5-d129defd0572") : secret "samples-operator-tls" not found
Sep 14 11:01:59.630872 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:01:59.630839    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hh9kh"]
Sep 14 11:01:59.635560 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:01:59.635533    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e805c86_98b1_4bc0_b457_b1100590d637.slice/crio-1a8b8a0558f9ccc2838d2c95d73b204514195fa9318dd294a38926c2f3dce7e3 WatchSource:0}: Error finding container 1a8b8a0558f9ccc2838d2c95d73b204514195fa9318dd294a38926c2f3dce7e3: Status 404 returned error can't find the container with id 1a8b8a0558f9ccc2838d2c95d73b204514195fa9318dd294a38926c2f3dce7e3
Sep 14 11:02:00.180082 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:00.180041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hnnpq" event={"ID":"cbefa077-d861-42b7-be43-c4c3b0bdfaba","Type":"ContainerStarted","Data":"96b45f9e0aa2d2f3d9b7430fe3d419ec596ce6e1284200388395c8bf69e9e62a"}
Sep 14 11:02:00.181462 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:00.181426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" event={"ID":"6e805c86-98b1-4bc0-b457-b1100590d637","Type":"ContainerStarted","Data":"1a8b8a0558f9ccc2838d2c95d73b204514195fa9318dd294a38926c2f3dce7e3"}
Sep 14 11:02:00.182669 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:00.182648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh" event={"ID":"f1d017a4-92d9-4e85-9d7e-bb8f64c2265a","Type":"ContainerStarted","Data":"1a804e03fe8108d5d4b10bd78a24a581d6f5cc36768510bc9d2243e85623b131"}
Sep 14 11:02:00.607039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:00.606997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:02:00.607222 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:00.607094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:00.607222 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:00.607211    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:00.607350 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:00.607274    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls podName:cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:02.607254954 +0000 UTC m=+71.229808162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gl2m9" (UID: "cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96") : secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:00.607781 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:00.607697    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 11:02:00.607781 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:00.607756    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls podName:9ce0c132-d207-4e7a-bad5-d129defd0572 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:02.607738901 +0000 UTC m=+71.230292106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4v88z" (UID: "9ce0c132-d207-4e7a-bad5-d129defd0572") : secret "samples-operator-tls" not found
Sep 14 11:02:02.622476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:02.622431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:02:02.622939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:02.622514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:02.622939 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:02.622604    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 11:02:02.622939 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:02.622616    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:02.622939 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:02.622710    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls podName:9ce0c132-d207-4e7a-bad5-d129defd0572 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.622687093 +0000 UTC m=+75.245240290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4v88z" (UID: "9ce0c132-d207-4e7a-bad5-d129defd0572") : secret "samples-operator-tls" not found
Sep 14 11:02:02.622939 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:02.622771    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls podName:cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.622738938 +0000 UTC m=+75.245292159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gl2m9" (UID: "cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96") : secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:05.196533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.196488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:05.199544 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.199503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jrkpt" event={"ID":"78e7293f-aa46-4991-910b-60558d5098bb","Type":"ContainerStarted","Data":"3a77b94245587157e968ddd4a086dcddb9744c9426609888cd07fd30718625b0"}
Sep 14 11:02:05.199734 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.199705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:02:05.201133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.201102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7sw9" event={"ID":"82f1808b-0670-4a82-8c87-39180882715e","Type":"ContainerStarted","Data":"2c8d103d588e6960547d8cfaa143d3b7a110b2fb409e07eda0d9d027274e8520"}
Sep 14 11:02:05.202748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.202725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hnnpq" event={"ID":"cbefa077-d861-42b7-be43-c4c3b0bdfaba","Type":"ContainerStarted","Data":"63c17f254aa874068fc099ed019b96062f280114290fd8331ade13411c88bfb2"}
Sep 14 11:02:05.204381 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.204363    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/0.log"
Sep 14 11:02:05.204488 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.204420    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e805c86-98b1-4bc0-b457-b1100590d637" containerID="e65de056d008ea0e62b4447eaa1351fae6aa3af6299c5bdf4b527debaba7b9fd" exitCode=255
Sep 14 11:02:05.204488 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.204479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" event={"ID":"6e805c86-98b1-4bc0-b457-b1100590d637","Type":"ContainerDied","Data":"e65de056d008ea0e62b4447eaa1351fae6aa3af6299c5bdf4b527debaba7b9fd"}
Sep 14 11:02:05.204640 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.204626    2563 scope.go:117] "RemoveContainer" containerID="e65de056d008ea0e62b4447eaa1351fae6aa3af6299c5bdf4b527debaba7b9fd"
Sep 14 11:02:05.206061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.206035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh" event={"ID":"f1d017a4-92d9-4e85-9d7e-bb8f64c2265a","Type":"ContainerStarted","Data":"d9aab08c468221ba8cc9d4d3705c6cc8c8acb3020994cb73e694224f163cf440"}
Sep 14 11:02:05.216982 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.216930    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jrkpt" podStartSLOduration=66.570427024 podStartE2EDuration="1m13.216910661s" totalImagesPullingTime="6.646483637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:05.216849304 +0000 UTC m=+73.839402519" watchObservedRunningTime="2026-09-14 11:02:05.216910661 +0000 UTC m=+73.839463876"
Sep 14 11:02:05.232581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.232522    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x7sw9" podStartSLOduration=66.466144466 podStartE2EDuration="1m13.232507897s" totalImagesPullingTime="6.766363431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:05.232076874 +0000 UTC m=+73.854630087" watchObservedRunningTime="2026-09-14 11:02:05.232507897 +0000 UTC m=+73.855061105"
Sep 14 11:02:05.248584 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.248541    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-hnnpq" podStartSLOduration=2.010917864 podStartE2EDuration="7.248524968s" totalImagesPullingTime="5.237607104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:05.248129962 +0000 UTC m=+73.870683201" watchObservedRunningTime="2026-09-14 11:02:05.248524968 +0000 UTC m=+73.871078181"
Sep 14 11:02:05.283986 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.283716    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-6gwgh" podStartSLOduration=1.927009953 podStartE2EDuration="7.283696486s" totalImagesPullingTime="5.356686533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:05.282536453 +0000 UTC m=+73.905089668" watchObservedRunningTime="2026-09-14 11:02:05.283696486 +0000 UTC m=+73.906249699"
Sep 14 11:02:05.289087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.288665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:05.431849 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.431812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:05.533694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.533611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:05.632271 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.632238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:05.694445 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.694413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:05.890936 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:05.890905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:06.010344 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.010297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:06.051211 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.051170    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"]
Sep 14 11:02:06.054621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.054591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.057695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.057662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k7rl9\""
Sep 14 11:02:06.058007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.057985    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 14 11:02:06.058287 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.058268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 14 11:02:06.068098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.068070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"]
Sep 14 11:02:06.099679 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.099643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:06.102041 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.102015    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zs5cm"]
Sep 14 11:02:06.105140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.105117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.107366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.107348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 11:02:06.107498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.107361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nzc8q\""
Sep 14 11:02:06.107498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.107354    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 11:02:06.117080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.117052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zs5cm"]
Sep 14 11:02:06.152829 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.152741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.152829 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.152798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8a10276-9416-4b01-b3e6-f17e67e44f04-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.153055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.152849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b851c979-1608-4c20-ac00-d0b030e36757-crio-socket\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.153055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.152908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.153055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.152930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b851c979-1608-4c20-ac00-d0b030e36757-data-volume\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.153055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.153013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.153196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.153059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58t62\" (UniqueName: \"kubernetes.io/projected/b851c979-1608-4c20-ac00-d0b030e36757-kube-api-access-58t62\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.211175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.211146    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:02:06.211604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.211585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/0.log"
Sep 14 11:02:06.211660 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.211624    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e805c86-98b1-4bc0-b457-b1100590d637" containerID="fc22f746043fd25acd3e1a531d351fb84234991d4abffefa9bc7f3a0d69e0db0" exitCode=255
Sep 14 11:02:06.211698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.211670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" event={"ID":"6e805c86-98b1-4bc0-b457-b1100590d637","Type":"ContainerDied","Data":"fc22f746043fd25acd3e1a531d351fb84234991d4abffefa9bc7f3a0d69e0db0"}
Sep 14 11:02:06.211746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.211731    2563 scope.go:117] "RemoveContainer" containerID="e65de056d008ea0e62b4447eaa1351fae6aa3af6299c5bdf4b527debaba7b9fd"
Sep 14 11:02:06.211991 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.211975    2563 scope.go:117] "RemoveContainer" containerID="fc22f746043fd25acd3e1a531d351fb84234991d4abffefa9bc7f3a0d69e0db0"
Sep 14 11:02:06.212191 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.212170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hh9kh_openshift-console-operator(6e805c86-98b1-4bc0-b457-b1100590d637)\"" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" podUID="6e805c86-98b1-4bc0-b457-b1100590d637"
Sep 14 11:02:06.253565 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.253527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b851c979-1608-4c20-ac00-d0b030e36757-data-volume\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.253761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.253577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.253761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.253644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58t62\" (UniqueName: \"kubernetes.io/projected/b851c979-1608-4c20-ac00-d0b030e36757-kube-api-access-58t62\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.253761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.253723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.253926 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.253786    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 11:02:06.253926 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.253866    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.753846409 +0000 UTC m=+75.376399615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : configmap "kube-rbac-proxy" not found
Sep 14 11:02:06.254044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.253922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b851c979-1608-4c20-ac00-d0b030e36757-data-volume\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.254044 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.254021    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 14 11:02:06.254143 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.254075    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert podName:c8a10276-9416-4b01-b3e6-f17e67e44f04 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.754058604 +0000 UTC m=+75.376611796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-25w8h" (UID: "c8a10276-9416-4b01-b3e6-f17e67e44f04") : secret "networking-console-plugin-cert" not found
Sep 14 11:02:06.254143 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.254111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8a10276-9416-4b01-b3e6-f17e67e44f04-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.254143 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.254133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b851c979-1608-4c20-ac00-d0b030e36757-crio-socket\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.254289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.254210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b851c979-1608-4c20-ac00-d0b030e36757-crio-socket\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.254340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.254279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.254456 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.254439    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:06.254522 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.254484    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.754472823 +0000 UTC m=+75.377026020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:06.254781 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.254763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8a10276-9416-4b01-b3e6-f17e67e44f04-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.266860 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.266830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58t62\" (UniqueName: \"kubernetes.io/projected/b851c979-1608-4c20-ac00-d0b030e36757-kube-api-access-58t62\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.456692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.456599    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"]
Sep 14 11:02:06.460939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.460915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"
Sep 14 11:02:06.463699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.463668    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8gzs7\""
Sep 14 11:02:06.463814 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.463697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 14 11:02:06.463814 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.463706    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 14 11:02:06.470692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.470666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"]
Sep 14 11:02:06.508608 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.508569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:06.556568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.556524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8z6\" (UniqueName: \"kubernetes.io/projected/2cc4a700-e25d-4a70-8dd4-5dcd3b134989-kube-api-access-zl8z6\") pod \"migrator-8b4d697c5-btl9j\" (UID: \"2cc4a700-e25d-4a70-8dd4-5dcd3b134989\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"
Sep 14 11:02:06.627253 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.627223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:06.657471 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.657434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8z6\" (UniqueName: \"kubernetes.io/projected/2cc4a700-e25d-4a70-8dd4-5dcd3b134989-kube-api-access-zl8z6\") pod \"migrator-8b4d697c5-btl9j\" (UID: \"2cc4a700-e25d-4a70-8dd4-5dcd3b134989\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"
Sep 14 11:02:06.657471 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.657472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:02:06.657694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.657540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:06.657694 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.657659    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:06.657694 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.657666    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 11:02:06.657808 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.657726    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls podName:cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:14.657705473 +0000 UTC m=+83.280258669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gl2m9" (UID: "cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96") : secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:06.657808 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.657744    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls podName:9ce0c132-d207-4e7a-bad5-d129defd0572 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:14.657734848 +0000 UTC m=+83.280288044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4v88z" (UID: "9ce0c132-d207-4e7a-bad5-d129defd0572") : secret "samples-operator-tls" not found
Sep 14 11:02:06.683149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.683117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8z6\" (UniqueName: \"kubernetes.io/projected/2cc4a700-e25d-4a70-8dd4-5dcd3b134989-kube-api-access-zl8z6\") pod \"migrator-8b4d697c5-btl9j\" (UID: \"2cc4a700-e25d-4a70-8dd4-5dcd3b134989\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"
Sep 14 11:02:06.758882 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.758777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.758882 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.758822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.758957    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.759005    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.758959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.759028    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.759009024 +0000 UTC m=+76.381562232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.759053    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.759057    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.759045948 +0000 UTC m=+76.381599140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : configmap "kube-rbac-proxy" not found
Sep 14 11:02:06.759115 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:06.759104    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert podName:c8a10276-9416-4b01-b3e6-f17e67e44f04 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.759093759 +0000 UTC m=+76.381646951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-25w8h" (UID: "c8a10276-9416-4b01-b3e6-f17e67e44f04") : secret "networking-console-plugin-cert" not found
Sep 14 11:02:06.770104 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:06.770073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"
Sep 14 11:02:07.043142 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.043060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j"]
Sep 14 11:02:07.046356 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:07.046326    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc4a700_e25d_4a70_8dd4_5dcd3b134989.slice/crio-76e9f028cbe2f9ef8dafc9d19f6586a81204f51864151449441c1b17bae5f31c WatchSource:0}: Error finding container 76e9f028cbe2f9ef8dafc9d19f6586a81204f51864151449441c1b17bae5f31c: Status 404 returned error can't find the container with id 76e9f028cbe2f9ef8dafc9d19f6586a81204f51864151449441c1b17bae5f31c
Sep 14 11:02:07.219468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.219423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j" event={"ID":"2cc4a700-e25d-4a70-8dd4-5dcd3b134989","Type":"ContainerStarted","Data":"76e9f028cbe2f9ef8dafc9d19f6586a81204f51864151449441c1b17bae5f31c"}
Sep 14 11:02:07.221054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.221032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:02:07.221404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.221368    2563 scope.go:117] "RemoveContainer" containerID="fc22f746043fd25acd3e1a531d351fb84234991d4abffefa9bc7f3a0d69e0db0"
Sep 14 11:02:07.221606 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.221574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hh9kh_openshift-console-operator(6e805c86-98b1-4bc0-b457-b1100590d637)\"" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" podUID="6e805c86-98b1-4bc0-b457-b1100590d637"
Sep 14 11:02:07.580634 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.580596    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lkw8g"]
Sep 14 11:02:07.583890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.583864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.586997 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.586968    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 14 11:02:07.587152 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.587024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 14 11:02:07.590122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.590093    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k4gll\""
Sep 14 11:02:07.590122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.590120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 14 11:02:07.598899 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.598866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 14 11:02:07.599882 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.599852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lkw8g"]
Sep 14 11:02:07.645078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.645040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:07.668138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.668097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b779edb7-e0a6-4151-b386-18c15676e7c5-signing-cabundle\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.668306 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.668167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqtd\" (UniqueName: \"kubernetes.io/projected/b779edb7-e0a6-4151-b386-18c15676e7c5-kube-api-access-9sqtd\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.668368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.668345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b779edb7-e0a6-4151-b386-18c15676e7c5-signing-key\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.743853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.743815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:07.769015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.768977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:07.769184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.769031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b779edb7-e0a6-4151-b386-18c15676e7c5-signing-cabundle\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.769184 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.769134    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 11:02:07.769301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.769189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqtd\" (UniqueName: \"kubernetes.io/projected/b779edb7-e0a6-4151-b386-18c15676e7c5-kube-api-access-9sqtd\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.769301 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.769213    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.769191741 +0000 UTC m=+78.391744940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : configmap "kube-rbac-proxy" not found
Sep 14 11:02:07.769450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.769301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:07.769450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.769354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:07.769450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.769428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b779edb7-e0a6-4151-b386-18c15676e7c5-signing-key\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.769616 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.769460    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:07.769616 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.769521    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.769502629 +0000 UTC m=+78.392055824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:07.769616 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.769547    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 14 11:02:07.769616 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:07.769607    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert podName:c8a10276-9416-4b01-b3e6-f17e67e44f04 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.769581042 +0000 UTC m=+78.392134240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-25w8h" (UID: "c8a10276-9416-4b01-b3e6-f17e67e44f04") : secret "networking-console-plugin-cert" not found
Sep 14 11:02:07.769976 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.769953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b779edb7-e0a6-4151-b386-18c15676e7c5-signing-cabundle\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.772259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.772238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b779edb7-e0a6-4151-b386-18c15676e7c5-signing-key\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.778198 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.778171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqtd\" (UniqueName: \"kubernetes.io/projected/b779edb7-e0a6-4151-b386-18c15676e7c5-kube-api-access-9sqtd\") pod \"service-ca-568869fdc5-lkw8g\" (UID: \"b779edb7-e0a6-4151-b386-18c15676e7c5\") " pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.894951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.894891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-lkw8g"
Sep 14 11:02:07.947733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:07.947703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d8dg5_2c60ef6d-94b5-4a01-afb1-99c1f6e99d41/dns-node-resolver/0.log"
Sep 14 11:02:08.225946 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:08.225915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j" event={"ID":"2cc4a700-e25d-4a70-8dd4-5dcd3b134989","Type":"ContainerStarted","Data":"0f4c20f1f733e5b076cb88179453a9f1c9a4ebbed95e0c98abf1cbc8f45e5508"}
Sep 14 11:02:08.339166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:08.339131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lkw8g"]
Sep 14 11:02:08.344269 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:08.344235    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb779edb7_e0a6_4151_b386_18c15676e7c5.slice/crio-e55530dc42da1ef03bbcc33a4cd2b183f71abca6534f4bc984642ece1b1edf4a WatchSource:0}: Error finding container e55530dc42da1ef03bbcc33a4cd2b183f71abca6534f4bc984642ece1b1edf4a: Status 404 returned error can't find the container with id e55530dc42da1ef03bbcc33a4cd2b183f71abca6534f4bc984642ece1b1edf4a
Sep 14 11:02:08.653064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:08.653024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:09.146259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.146231    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7psjm_0a6d424e-e203-4fae-9ff5-db0982a2f37b/node-ca/0.log"
Sep 14 11:02:09.232601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.232536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j" event={"ID":"2cc4a700-e25d-4a70-8dd4-5dcd3b134989","Type":"ContainerStarted","Data":"80883f396a788d1f5f8db3908e571acea4d5f4cd4a7bd10e73302156808d4779"}
Sep 14 11:02:09.234020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.233991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-lkw8g" event={"ID":"b779edb7-e0a6-4151-b386-18c15676e7c5","Type":"ContainerStarted","Data":"e55530dc42da1ef03bbcc33a4cd2b183f71abca6534f4bc984642ece1b1edf4a"}
Sep 14 11:02:09.250566 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.250505    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-btl9j" podStartSLOduration=2.186399659 podStartE2EDuration="3.250483871s" totalImagesPullingTime="1.064084212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:09.249698481 +0000 UTC m=+77.872251695" watchObservedRunningTime="2026-09-14 11:02:09.250483871 +0000 UTC m=+77.873037086"
Sep 14 11:02:09.349897 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.349848    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:02:09.350063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.349935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:02:09.350301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.350287    2563 scope.go:117] "RemoveContainer" containerID="fc22f746043fd25acd3e1a531d351fb84234991d4abffefa9bc7f3a0d69e0db0"
Sep 14 11:02:09.350483 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.350464    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hh9kh_openshift-console-operator(6e805c86-98b1-4bc0-b457-b1100590d637)\"" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" podUID="6e805c86-98b1-4bc0-b457-b1100590d637"
Sep 14 11:02:09.788689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.788648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:09.788853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.788751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:09.788853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.788783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:09.788853 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.788791    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 11:02:09.788998 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.788874    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.788854162 +0000 UTC m=+82.411407370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : configmap "kube-rbac-proxy" not found
Sep 14 11:02:09.788998 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.788876    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 14 11:02:09.788998 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.788902    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:09.788998 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.788927    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert podName:c8a10276-9416-4b01-b3e6-f17e67e44f04 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.788911899 +0000 UTC m=+82.411465094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-25w8h" (UID: "c8a10276-9416-4b01-b3e6-f17e67e44f04") : secret "networking-console-plugin-cert" not found
Sep 14 11:02:09.788998 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:09.788945    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.788933181 +0000 UTC m=+82.411486396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:09.947408 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:09.947362    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9lhw2_02c8187f-5f7e-4860-a58f-65496fd89b31/serve-healthcheck-canary/0.log"
Sep 14 11:02:10.238443 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:10.238407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-lkw8g" event={"ID":"b779edb7-e0a6-4151-b386-18c15676e7c5","Type":"ContainerStarted","Data":"691d1b5eaaf9b6cf31b9b98c6ca976892908ee9faac10b411bab09ae0ac26667"}
Sep 14 11:02:10.258315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:10.258257    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-lkw8g" podStartSLOduration=1.652489246 podStartE2EDuration="3.258241849s" totalImagesPullingTime="1.605752603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:10.257738855 +0000 UTC m=+78.880292070" watchObservedRunningTime="2026-09-14 11:02:10.258241849 +0000 UTC m=+78.880795063"
Sep 14 11:02:10.986063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:10.986026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:11.081980 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:11.081941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:11.186961 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:11.186915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:12.841533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:12.841488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:12.908267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:12.908238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:13.054743 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:13.054710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:13.458338 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:13.458305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:13.826165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:13.826069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:13.826165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:13.826137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:13.826352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:13.826200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:13.826352 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:13.826249    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:13.826352 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:13.826287    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 14 11:02:13.826352 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:13.826322    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 11:02:13.826352 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:13.826331    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.826309916 +0000 UTC m=+90.448863130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : secret "insights-runtime-extractor-tls" not found
Sep 14 11:02:13.826352 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:13.826351    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert podName:c8a10276-9416-4b01-b3e6-f17e67e44f04 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.826341319 +0000 UTC m=+90.448894516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-25w8h" (UID: "c8a10276-9416-4b01-b3e6-f17e67e44f04") : secret "networking-console-plugin-cert" not found
Sep 14 11:02:13.826642 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:13.826367    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm podName:b851c979-1608-4c20-ac00-d0b030e36757 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.826358509 +0000 UTC m=+90.448911708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zs5cm" (UID: "b851c979-1608-4c20-ac00-d0b030e36757") : configmap "kube-rbac-proxy" not found
Sep 14 11:02:14.732747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:14.732611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:02:14.732747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:14.732693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:14.733161 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:14.732791    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:14.733161 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:14.732858    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls podName:cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:30.732840594 +0000 UTC m=+99.355393800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gl2m9" (UID: "cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96") : secret "cluster-monitoring-operator-tls" not found
Sep 14 11:02:14.735181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:14.735158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ce0c132-d207-4e7a-bad5-d129defd0572-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4v88z\" (UID: \"9ce0c132-d207-4e7a-bad5-d129defd0572\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:02:14.834559 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:14.834528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"
Sep 14 11:02:15.105761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:15.105714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z"]
Sep 14 11:02:15.165259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:15.165235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:15.256586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:15.256549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z" event={"ID":"9ce0c132-d207-4e7a-bad5-d129defd0572","Type":"ContainerStarted","Data":"d9383cd26bf1392953b4af8f1f957607bed912a86c5600e898bdc8d158a75f6e"}
Sep 14 11:02:15.265491 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:15.265462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:15.368332 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:15.368249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:15.466842 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:15.466802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:16.543640 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:16.543602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:16.645151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:16.645118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:16.856459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:16.856421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:17.453204 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:17.453149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:17.515385 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:17.515356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:17.652728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:17.652680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:17.852854 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:17.852818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:18.055438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:18.055406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:18.269726 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:18.269689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z" event={"ID":"9ce0c132-d207-4e7a-bad5-d129defd0572","Type":"ContainerStarted","Data":"244ce46e0fa307e29e549d7903ed04c7186d45e2d58d9aeea804a6a42e800df3"}
Sep 14 11:02:18.269895 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:18.269747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z" event={"ID":"9ce0c132-d207-4e7a-bad5-d129defd0572","Type":"ContainerStarted","Data":"5350ed9d532229b9dd8e7709cdd6134c89b16cae200ca9da9eb1a5b8cbc0fd12"}
Sep 14 11:02:18.293377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:18.293326    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4v88z" podStartSLOduration=17.337564187 podStartE2EDuration="20.293307904s" totalImagesPullingTime="2.955743717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:18.292806096 +0000 UTC m=+86.915359311" watchObservedRunningTime="2026-09-14 11:02:18.293307904 +0000 UTC m=+86.915861118"
Sep 14 11:02:18.670369 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:18.670339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:19.068623 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:19.068535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:20.851837 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:20.851806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:20.912827 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:20.912798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:21.056615 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.056582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:21.259323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.259214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:21.457735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.457647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:21.653873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.653841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:21.887834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.887802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:21.888240 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.887843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:21.888240 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.887884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:21.888466 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.888448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b851c979-1608-4c20-ac00-d0b030e36757-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:21.890219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.890196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b851c979-1608-4c20-ac00-d0b030e36757-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zs5cm\" (UID: \"b851c979-1608-4c20-ac00-d0b030e36757\") " pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:21.890382 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.890361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8a10276-9416-4b01-b3e6-f17e67e44f04-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-25w8h\" (UID: \"c8a10276-9416-4b01-b3e6-f17e67e44f04\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:21.966769 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:21.966684    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"
Sep 14 11:02:22.016223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:22.015094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zs5cm"
Sep 14 11:02:22.240327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:22.240244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h"]
Sep 14 11:02:22.243653 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:22.243625    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a10276_9416_4b01_b3e6_f17e67e44f04.slice/crio-e3404b5ef90239b5270260118b758e5df974d7a6880df6250c3f7e24c79f3cab WatchSource:0}: Error finding container e3404b5ef90239b5270260118b758e5df974d7a6880df6250c3f7e24c79f3cab: Status 404 returned error can't find the container with id e3404b5ef90239b5270260118b758e5df974d7a6880df6250c3f7e24c79f3cab
Sep 14 11:02:22.282241 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:22.282201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h" event={"ID":"c8a10276-9416-4b01-b3e6-f17e67e44f04","Type":"ContainerStarted","Data":"e3404b5ef90239b5270260118b758e5df974d7a6880df6250c3f7e24c79f3cab"}
Sep 14 11:02:22.293038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:22.293007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zs5cm"]
Sep 14 11:02:22.297309 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:22.297280    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb851c979_1608_4c20_ac00_d0b030e36757.slice/crio-a4a135eedface50a838e1edac06d68ee0538a129f064ee07b87d8dd656018d82 WatchSource:0}: Error finding container a4a135eedface50a838e1edac06d68ee0538a129f064ee07b87d8dd656018d82: Status 404 returned error can't find the container with id a4a135eedface50a838e1edac06d68ee0538a129f064ee07b87d8dd656018d82
Sep 14 11:02:22.912742 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:22.912714    2563 scope.go:117] "RemoveContainer" containerID="fc22f746043fd25acd3e1a531d351fb84234991d4abffefa9bc7f3a0d69e0db0"
Sep 14 11:02:23.288635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.288534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zs5cm" event={"ID":"b851c979-1608-4c20-ac00-d0b030e36757","Type":"ContainerStarted","Data":"7b580ef4554e9070a2f8d472d0bb18de8e3a76986848804baedf6001f7d1b620"}
Sep 14 11:02:23.288635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.288616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zs5cm" event={"ID":"b851c979-1608-4c20-ac00-d0b030e36757","Type":"ContainerStarted","Data":"e5da34487ed41a1b6b097e9dc2848396a6e3283d7215d61009ce6d0e373e6636"}
Sep 14 11:02:23.288838 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.288651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zs5cm" event={"ID":"b851c979-1608-4c20-ac00-d0b030e36757","Type":"ContainerStarted","Data":"a4a135eedface50a838e1edac06d68ee0538a129f064ee07b87d8dd656018d82"}
Sep 14 11:02:23.290357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.290338    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:02:23.290480 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.290380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" event={"ID":"6e805c86-98b1-4bc0-b457-b1100590d637","Type":"ContainerStarted","Data":"b1eb5a0758ab15f7c5da9efd84e75a4e67f6252a726ec1adefb5024a552acc05"}
Sep 14 11:02:23.290672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.290647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:02:23.656374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.656292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh"
Sep 14 11:02:23.681367 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.680315    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-hh9kh" podStartSLOduration=19.563197485 podStartE2EDuration="24.680289069s" totalImagesPullingTime="5.117091584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:23.32186211 +0000 UTC m=+91.944415322" watchObservedRunningTime="2026-09-14 11:02:23.680289069 +0000 UTC m=+92.302842348"
Sep 14 11:02:23.851839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.851807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:23.953040 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:23.952959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:24.017847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:24.017819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:24.139955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:24.139918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:25.303054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:25.303013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h" event={"ID":"c8a10276-9416-4b01-b3e6-f17e67e44f04","Type":"ContainerStarted","Data":"63b03707e70dcda6a1976f1ffa57e97a70bfb7f4be7f7ddf159461cb13916d81"}
Sep 14 11:02:25.305352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:25.305325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zs5cm" event={"ID":"b851c979-1608-4c20-ac00-d0b030e36757","Type":"ContainerStarted","Data":"d39d70fc2208924b8f5bdec76c432db9e4ac8f76c13631dd9032b754fe08aecb"}
Sep 14 11:02:25.320854 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:25.320804    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-25w8h" podStartSLOduration=17.119269192 podStartE2EDuration="19.320791113s" totalImagesPullingTime="2.201521921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:25.320784055 +0000 UTC m=+93.943337270" watchObservedRunningTime="2026-09-14 11:02:25.320791113 +0000 UTC m=+93.943344347"
Sep 14 11:02:25.341828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:25.341782    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zs5cm" podStartSLOduration=17.302807087 podStartE2EDuration="19.341766462s" totalImagesPullingTime="2.038959375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:02:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:25.340904343 +0000 UTC m=+93.963457558" watchObservedRunningTime="2026-09-14 11:02:25.341766462 +0000 UTC m=+93.964319680"
Sep 14 11:02:25.999600 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:25.999569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:26.062864 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:26.062832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:26.167435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:26.167380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:26.654610 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:26.654579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:27.499273 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:27.499243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:27.603404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:27.603354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:27.700182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:27.700148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:29.657941 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:29.657897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:02:29.660281 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:29.660254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/449a4fc3-67e5-41ad-b69f-33551d650e3a-metrics-tls\") pod \"dns-default-745q5\" (UID: \"449a4fc3-67e5-41ad-b69f-33551d650e3a\") " pod="openshift-dns/dns-default-745q5"
Sep 14 11:02:29.677900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:29.677870    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpbn2\""
Sep 14 11:02:29.684969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:29.684953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-745q5"
Sep 14 11:02:29.971759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:29.971683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-745q5"]
Sep 14 11:02:29.974708 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:29.974681    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449a4fc3_67e5_41ad_b69f_33551d650e3a.slice/crio-a5a60bbac7d5089565db53899edd47a3155c25cfef42fd9373eb73a5b0dec376 WatchSource:0}: Error finding container a5a60bbac7d5089565db53899edd47a3155c25cfef42fd9373eb73a5b0dec376: Status 404 returned error can't find the container with id a5a60bbac7d5089565db53899edd47a3155c25cfef42fd9373eb73a5b0dec376
Sep 14 11:02:30.323491 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:30.323410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-745q5" event={"ID":"449a4fc3-67e5-41ad-b69f-33551d650e3a","Type":"ContainerStarted","Data":"a5a60bbac7d5089565db53899edd47a3155c25cfef42fd9373eb73a5b0dec376"}
Sep 14 11:02:30.755049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:30.755014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:30.767229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:30.767200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:30.769735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:30.769712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gl2m9\" (UID: \"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:30.810088 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:30.810052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:31.043107 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:31.043024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"
Sep 14 11:02:31.331865 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:31.331773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9"]
Sep 14 11:02:31.336308 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:31.336275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7758e7_57f1_4ab8_b7c0_bf6d75e2dc96.slice/crio-539b7df98689028bd311f21ee430b3ed3addc1b5bb70552e36b474fa6e852c54 WatchSource:0}: Error finding container 539b7df98689028bd311f21ee430b3ed3addc1b5bb70552e36b474fa6e852c54: Status 404 returned error can't find the container with id 539b7df98689028bd311f21ee430b3ed3addc1b5bb70552e36b474fa6e852c54
Sep 14 11:02:32.173722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.173698    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5647bc96b4-bh85d"]
Sep 14 11:02:32.176867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.176849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.179217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179197    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 11:02:32.179329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjn2\" (UniqueName: \"kubernetes.io/projected/3acc29cb-b262-4eaf-9c20-59d6249cf99f-kube-api-access-smjn2\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.179329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-oauth-config\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.179329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179295    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 11:02:32.179329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 11:02:32.179553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-serving-cert\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.179553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 11:02:32.179553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-config\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.179553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-oauth-serving-cert\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.179553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.179468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-service-ca\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.180027 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.180004    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 11:02:32.180134 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.180116    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 11:02:32.180206 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.180014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 11:02:32.180327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.180311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-77gsx\""
Sep 14 11:02:32.190530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.190503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5647bc96b4-bh85d"]
Sep 14 11:02:32.280152 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-config\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.280283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-oauth-serving-cert\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.280283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-service-ca\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.280283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjn2\" (UniqueName: \"kubernetes.io/projected/3acc29cb-b262-4eaf-9c20-59d6249cf99f-kube-api-access-smjn2\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.280504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-oauth-config\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.280504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-serving-cert\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.280981 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.280954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-config\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.281252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.281204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-oauth-serving-cert\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.281252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.281239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-service-ca\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.283091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.283060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-serving-cert\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.284167 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.284141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-oauth-config\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.291463 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.291436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjn2\" (UniqueName: \"kubernetes.io/projected/3acc29cb-b262-4eaf-9c20-59d6249cf99f-kube-api-access-smjn2\") pod \"console-5647bc96b4-bh85d\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") " pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.333743 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.333709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9" event={"ID":"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96","Type":"ContainerStarted","Data":"539b7df98689028bd311f21ee430b3ed3addc1b5bb70552e36b474fa6e852c54"}
Sep 14 11:02:32.335587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.335557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-745q5" event={"ID":"449a4fc3-67e5-41ad-b69f-33551d650e3a","Type":"ContainerStarted","Data":"55de5d8035f5fb3924c9563f8c4d655381043d535ebeeca71f4315fa1f08a0f9"}
Sep 14 11:02:32.496101 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.496022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:32.783612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:32.783450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5647bc96b4-bh85d"]
Sep 14 11:02:32.786421 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:32.786366    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acc29cb_b262_4eaf_9c20_59d6249cf99f.slice/crio-5c5f8558199fd0dceb39eba78c1475833b6696ef4ab185ceccf2e8d714b406cd WatchSource:0}: Error finding container 5c5f8558199fd0dceb39eba78c1475833b6696ef4ab185ceccf2e8d714b406cd: Status 404 returned error can't find the container with id 5c5f8558199fd0dceb39eba78c1475833b6696ef4ab185ceccf2e8d714b406cd
Sep 14 11:02:33.344536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.344496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-745q5" event={"ID":"449a4fc3-67e5-41ad-b69f-33551d650e3a","Type":"ContainerStarted","Data":"7a93e94f55671234f1985a37532c10b215250ca2aa1752b98a58737ddb21c6cd"}
Sep 14 11:02:33.344990 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.344620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-745q5"
Sep 14 11:02:33.345713 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.345683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647bc96b4-bh85d" event={"ID":"3acc29cb-b262-4eaf-9c20-59d6249cf99f","Type":"ContainerStarted","Data":"5c5f8558199fd0dceb39eba78c1475833b6696ef4ab185ceccf2e8d714b406cd"}
Sep 14 11:02:33.363694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.363636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-745q5" podStartSLOduration=66.175559339 podStartE2EDuration="1m8.363613131s" totalImagesPullingTime="2.188053792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:33.362156228 +0000 UTC m=+101.984709441" watchObservedRunningTime="2026-09-14 11:02:33.363613131 +0000 UTC m=+101.986166346"
Sep 14 11:02:33.681711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.681674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:33.846650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.846607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:33.860890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.860859    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"]
Sep 14 11:02:33.866054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.866037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:33.868431 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.868384    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 14 11:02:33.868431 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.868437    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rdqzr\""
Sep 14 11:02:33.873903 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.873876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"]
Sep 14 11:02:33.895297 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.895268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ecf4d8dd-3cf4-4464-98ed-df18607beb2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-k2wm5\" (UID: \"ecf4d8dd-3cf4-4464-98ed-df18607beb2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:33.995981 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:33.995899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ecf4d8dd-3cf4-4464-98ed-df18607beb2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-k2wm5\" (UID: \"ecf4d8dd-3cf4-4464-98ed-df18607beb2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:33.996138 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:33.996074    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 14 11:02:33.996186 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:33.996152    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecf4d8dd-3cf4-4464-98ed-df18607beb2e-tls-certificates podName:ecf4d8dd-3cf4-4464-98ed-df18607beb2e nodeName:}" failed. No retries permitted until 2026-09-14 11:02:34.496131141 +0000 UTC m=+103.118684339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ecf4d8dd-3cf4-4464-98ed-df18607beb2e-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-k2wm5" (UID: "ecf4d8dd-3cf4-4464-98ed-df18607beb2e") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 14 11:02:34.172626 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:34.172594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:34.350597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:34.350561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9" event={"ID":"cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96","Type":"ContainerStarted","Data":"c190f5709ad37fc5e3c6db40fb57aff4b266567969ef8316cda4427fe6bf9f6b"}
Sep 14 11:02:34.369315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:34.369264    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gl2m9" podStartSLOduration=34.357505324 podStartE2EDuration="36.369246845s" totalImagesPullingTime="2.011741521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:34.367522806 +0000 UTC m=+102.990076033" watchObservedRunningTime="2026-09-14 11:02:34.369246845 +0000 UTC m=+102.991800327"
Sep 14 11:02:34.500245 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:34.500209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ecf4d8dd-3cf4-4464-98ed-df18607beb2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-k2wm5\" (UID: \"ecf4d8dd-3cf4-4464-98ed-df18607beb2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:34.502772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:34.502746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ecf4d8dd-3cf4-4464-98ed-df18607beb2e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-k2wm5\" (UID: \"ecf4d8dd-3cf4-4464-98ed-df18607beb2e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:34.775893 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:34.775808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:35.079353 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:35.079279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"]
Sep 14 11:02:35.082414 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:35.082376    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf4d8dd_3cf4_4464_98ed_df18607beb2e.slice/crio-31dad23bd3c78b462b2a49c230594b689b11fc3a0a7840b153cfef99f21219de WatchSource:0}: Error finding container 31dad23bd3c78b462b2a49c230594b689b11fc3a0a7840b153cfef99f21219de: Status 404 returned error can't find the container with id 31dad23bd3c78b462b2a49c230594b689b11fc3a0a7840b153cfef99f21219de
Sep 14 11:02:35.356930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:35.356880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5" event={"ID":"ecf4d8dd-3cf4-4464-98ed-df18607beb2e","Type":"ContainerStarted","Data":"31dad23bd3c78b462b2a49c230594b689b11fc3a0a7840b153cfef99f21219de"}
Sep 14 11:02:36.215051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:36.215016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jrkpt"
Sep 14 11:02:37.569862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:37.569825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:37.637179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:37.637146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:38.369473 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.369435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5" event={"ID":"ecf4d8dd-3cf4-4464-98ed-df18607beb2e","Type":"ContainerStarted","Data":"ea9b37086b338c117d2e09e6d0c58b0ae809fd78518f1433929f60497c28738a"}
Sep 14 11:02:38.369649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.369624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:38.371296 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.371263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647bc96b4-bh85d" event={"ID":"3acc29cb-b262-4eaf-9c20-59d6249cf99f","Type":"ContainerStarted","Data":"3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c"}
Sep 14 11:02:38.374967 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.374945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5"
Sep 14 11:02:38.403384 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.403328    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-k2wm5" podStartSLOduration=2.765011989 podStartE2EDuration="5.403313286s" totalImagesPullingTime="2.638301297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:38.388064005 +0000 UTC m=+107.010617219" watchObservedRunningTime="2026-09-14 11:02:38.403313286 +0000 UTC m=+107.025866499"
Sep 14 11:02:38.422212 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.422150    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5647bc96b4-bh85d" podStartSLOduration=1.128047886 podStartE2EDuration="6.422126969s" totalImagesPullingTime="5.294079083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:38.42127693 +0000 UTC m=+107.043830144" watchObservedRunningTime="2026-09-14 11:02:38.422126969 +0000 UTC m=+107.044680185"
Sep 14 11:02:38.883940 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.883905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:38.936561 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.936526    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"]
Sep 14 11:02:38.958270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.958243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"]
Sep 14 11:02:38.958438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.958339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:38.960875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.960626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 14 11:02:38.960875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.960704    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 11:02:38.960875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.960744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 14 11:02:38.961139 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.960972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qfghv\""
Sep 14 11:02:38.961139 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:38.960983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:39.037406 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.037353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.037406 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.037404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtt9\" (UniqueName: \"kubernetes.io/projected/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-kube-api-access-mdtt9\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.037676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.037455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.037676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.037528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.138275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.138179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.138275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.138242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.138513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.138292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtt9\" (UniqueName: \"kubernetes.io/projected/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-kube-api-access-mdtt9\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.138513 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:39.138403    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 14 11:02:39.138513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.138431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.138513 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:39.138475    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-tls podName:c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd nodeName:}" failed. No retries permitted until 2026-09-14 11:02:39.638453502 +0000 UTC m=+108.261006701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-km5rx" (UID: "c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd") : secret "prometheus-operator-tls" not found
Sep 14 11:02:39.138969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.138949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.140856 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.140833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.147039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.147015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtt9\" (UniqueName: \"kubernetes.io/projected/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-kube-api-access-mdtt9\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.250744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.250706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:39.454156 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.454075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:39.510016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.509977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:39.643080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.643043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.645573 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.645541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-km5rx\" (UID: \"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:39.763457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.763363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:39.824811 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.824780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:39.867949 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:39.867917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"
Sep 14 11:02:40.167645 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:40.167511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-km5rx"]
Sep 14 11:02:40.169877 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:40.169853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cdc9e5_5d03_486e_bc0c_0fccf2abeefd.slice/crio-80ae96cd2341aa89e4f76a3d6a7ce90d27f1f7d8a4fc45a0972dea150f114b1f WatchSource:0}: Error finding container 80ae96cd2341aa89e4f76a3d6a7ce90d27f1f7d8a4fc45a0972dea150f114b1f: Status 404 returned error can't find the container with id 80ae96cd2341aa89e4f76a3d6a7ce90d27f1f7d8a4fc45a0972dea150f114b1f
Sep 14 11:02:40.379792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:40.379752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx" event={"ID":"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd","Type":"ContainerStarted","Data":"80ae96cd2341aa89e4f76a3d6a7ce90d27f1f7d8a4fc45a0972dea150f114b1f"}
Sep 14 11:02:41.505046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:41.505014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:41.600593 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:41.600552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:41.703026 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:41.702943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:41.972669 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:41.972582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:42.070492 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.070437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:42.394530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.394496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx" event={"ID":"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd","Type":"ContainerStarted","Data":"270b05c37048ca35800015dddbc00f00957017f7985a2fd93a6c3a7bf7cd84d5"}
Sep 14 11:02:42.394728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.394555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx" event={"ID":"c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd","Type":"ContainerStarted","Data":"f58020c7818eb1a08c0967d5b2167e1b0d0dc0de6cc59066cd668fbd04cf3873"}
Sep 14 11:02:42.417451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.417384    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-km5rx" podStartSLOduration=3.194630903 podStartE2EDuration="4.417368342s" totalImagesPullingTime="1.222737439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:42.416191837 +0000 UTC m=+111.038745050" watchObservedRunningTime="2026-09-14 11:02:42.417368342 +0000 UTC m=+111.039921556"
Sep 14 11:02:42.496529 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.496478    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:42.496717 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.496600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:42.501766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.501737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:42.562130 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:42.562094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:43.353208 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:43.353175    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-745q5"
Sep 14 11:02:43.402166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:43.402141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:02:43.498447 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:43.498410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:43.568823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:43.568792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.045218 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.045187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.266969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.266937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.308884 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.308795    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"]
Sep 14 11:02:44.313097 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.313065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.316865 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.316829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 14 11:02:44.316865 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.316828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qnfcq\""
Sep 14 11:02:44.317153 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.317128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 14 11:02:44.332468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.332440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"]
Sep 14 11:02:44.333972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.333951    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"]
Sep 14 11:02:44.337716 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.337693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.342259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.342235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 14 11:02:44.343175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.343152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 14 11:02:44.343346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.343330    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4gq7j\""
Sep 14 11:02:44.343524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.343507    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 14 11:02:44.348747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.348725    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5bn8j"]
Sep 14 11:02:44.352095 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.352077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.356460 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.356438    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56dhl\""
Sep 14 11:02:44.356604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.356445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 11:02:44.356681 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.356657    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 11:02:44.357746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.356094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 11:02:44.359887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.359417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"]
Sep 14 11:02:44.375673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.375646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.383374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.383568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8wf\" (UniqueName: \"kubernetes.io/projected/8a797eb2-ec29-4d76-827b-19e913d423de-kube-api-access-nr8wf\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.383568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8a797eb2-ec29-4d76-827b-19e913d423de-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.383568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.383695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a797eb2-ec29-4d76-827b-19e913d423de-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.383695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.383695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.383695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.383695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g289l\" (UniqueName: \"kubernetes.io/projected/922ac942-96fd-4206-b6c6-25c61035427e-kube-api-access-g289l\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.383910 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.383741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922ac942-96fd-4206-b6c6-25c61035427e-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.472872 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.472836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.484727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-wtmp\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.484902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a797eb2-ec29-4d76-827b-19e913d423de-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.484902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-root\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.484902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-textfile\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.484902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.484902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.484902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g289l\" (UniqueName: \"kubernetes.io/projected/922ac942-96fd-4206-b6c6-25c61035427e-kube-api-access-g289l\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.484996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-sys\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca11ed89-1292-43c6-84fe-cb2035d1f52c-metrics-client-ca\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922ac942-96fd-4206-b6c6-25c61035427e-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-tls\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.485194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8wf\" (UniqueName: \"kubernetes.io/projected/8a797eb2-ec29-4d76-827b-19e913d423de-kube-api-access-nr8wf\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.485722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8a797eb2-ec29-4d76-827b-19e913d423de-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.485722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.485722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.485247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5km\" (UniqueName: \"kubernetes.io/projected/ca11ed89-1292-43c6-84fe-cb2035d1f52c-kube-api-access-nk5km\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.485722 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:44.485500    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 14 11:02:44.485722 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:44.485563    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-tls podName:922ac942-96fd-4206-b6c6-25c61035427e nodeName:}" failed. No retries permitted until 2026-09-14 11:02:44.985542529 +0000 UTC m=+113.608095727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-769cx" (UID: "922ac942-96fd-4206-b6c6-25c61035427e") : secret "openshift-state-metrics-tls" not found
Sep 14 11:02:44.486257 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.486233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922ac942-96fd-4206-b6c6-25c61035427e-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.486666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.486647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a797eb2-ec29-4d76-827b-19e913d423de-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.486804 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.486777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.486934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.486914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8a797eb2-ec29-4d76-827b-19e913d423de-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.488554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.488529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.488789 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.488769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.488944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.488926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a797eb2-ec29-4d76-827b-19e913d423de-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.498650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.498610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g289l\" (UniqueName: \"kubernetes.io/projected/922ac942-96fd-4206-b6c6-25c61035427e-kube-api-access-g289l\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.501486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.501454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8wf\" (UniqueName: \"kubernetes.io/projected/8a797eb2-ec29-4d76-827b-19e913d423de-kube-api-access-nr8wf\") pod \"kube-state-metrics-59fbdd77d-zrc4g\" (UID: \"8a797eb2-ec29-4d76-827b-19e913d423de\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.576198 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.576083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.586278 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-tls\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586447 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:44.586294    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 11:02:44.586447 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:02:44.586362    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-tls podName:ca11ed89-1292-43c6-84fe-cb2035d1f52c nodeName:}" failed. No retries permitted until 2026-09-14 11:02:45.086345585 +0000 UTC m=+113.708898777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-tls") pod "node-exporter-5bn8j" (UID: "ca11ed89-1292-43c6-84fe-cb2035d1f52c") : secret "node-exporter-tls" not found
Sep 14 11:02:44.586447 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5km\" (UniqueName: \"kubernetes.io/projected/ca11ed89-1292-43c6-84fe-cb2035d1f52c-kube-api-access-nk5km\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586447 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-wtmp\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-root\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-textfile\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-sys\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca11ed89-1292-43c6-84fe-cb2035d1f52c-metrics-client-ca\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.586672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.586600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.587564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.587004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-wtmp\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.587564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.587021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-sys\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.587564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.587191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca11ed89-1292-43c6-84fe-cb2035d1f52c-root\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.587564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.587217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-textfile\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.587847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.587572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.587847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.587794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca11ed89-1292-43c6-84fe-cb2035d1f52c-metrics-client-ca\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.589557 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.589513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.594615 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.594594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5km\" (UniqueName: \"kubernetes.io/projected/ca11ed89-1292-43c6-84fe-cb2035d1f52c-kube-api-access-nk5km\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:44.649661 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.649623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"
Sep 14 11:02:44.712282 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.712247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.812353 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.812318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:44.928619 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.928592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g"]
Sep 14 11:02:44.930267 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:44.930245    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a797eb2_ec29_4d76_827b_19e913d423de.slice/crio-6590272abd2947992753c6380854107fca8e6d0b350455db91c70a0467408731 WatchSource:0}: Error finding container 6590272abd2947992753c6380854107fca8e6d0b350455db91c70a0467408731: Status 404 returned error can't find the container with id 6590272abd2947992753c6380854107fca8e6d0b350455db91c70a0467408731
Sep 14 11:02:44.990493 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.990455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:44.992839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:44.992812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922ac942-96fd-4206-b6c6-25c61035427e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-769cx\" (UID: \"922ac942-96fd-4206-b6c6-25c61035427e\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:45.091409 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.091353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-tls\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:45.093830 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.093803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca11ed89-1292-43c6-84fe-cb2035d1f52c-node-exporter-tls\") pod \"node-exporter-5bn8j\" (UID: \"ca11ed89-1292-43c6-84fe-cb2035d1f52c\") " pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:45.119960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.119928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:45.225592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.225502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"
Sep 14 11:02:45.263383 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.263342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bn8j"
Sep 14 11:02:45.279181 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:45.279137    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca11ed89_1292_43c6_84fe_cb2035d1f52c.slice/crio-166d5d9ddd4c870bb5be9e6d66a250c0817bafa621fc1e753a94d4eaa31b9524 WatchSource:0}: Error finding container 166d5d9ddd4c870bb5be9e6d66a250c0817bafa621fc1e753a94d4eaa31b9524: Status 404 returned error can't find the container with id 166d5d9ddd4c870bb5be9e6d66a250c0817bafa621fc1e753a94d4eaa31b9524
Sep 14 11:02:45.321792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.321756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:45.412786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.412743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bn8j" event={"ID":"ca11ed89-1292-43c6-84fe-cb2035d1f52c","Type":"ContainerStarted","Data":"166d5d9ddd4c870bb5be9e6d66a250c0817bafa621fc1e753a94d4eaa31b9524"}
Sep 14 11:02:45.414576 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.414528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g" event={"ID":"8a797eb2-ec29-4d76-827b-19e913d423de","Type":"ContainerStarted","Data":"6590272abd2947992753c6380854107fca8e6d0b350455db91c70a0467408731"}
Sep 14 11:02:45.421704 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.421654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:45.518349 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.518263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx"]
Sep 14 11:02:45.654163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:45.654127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:46.375207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.373195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:46.427564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.426909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g" event={"ID":"8a797eb2-ec29-4d76-827b-19e913d423de","Type":"ContainerStarted","Data":"f9628745c9758098b2e2dc8d668ef600066292a90c6d90ca6a9523be532d4ca5"}
Sep 14 11:02:46.427564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.426988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g" event={"ID":"8a797eb2-ec29-4d76-827b-19e913d423de","Type":"ContainerStarted","Data":"a6222b939c5f19488a8ea103a96ad1460fe3e9bee7db5bf622809b39f9313661"}
Sep 14 11:02:46.427564 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.427019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g" event={"ID":"8a797eb2-ec29-4d76-827b-19e913d423de","Type":"ContainerStarted","Data":"a88b198f9a0bdc96f453e7aaba6699c309f8b2144db657c39ca0c783dd62e9ce"}
Sep 14 11:02:46.431227 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.431163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx" event={"ID":"922ac942-96fd-4206-b6c6-25c61035427e","Type":"ContainerStarted","Data":"7c97a3568c372820e7a08b2fc832b8df62592e87ad6cc0ce7000a74a7b20cfa6"}
Sep 14 11:02:46.431613 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.431447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx" event={"ID":"922ac942-96fd-4206-b6c6-25c61035427e","Type":"ContainerStarted","Data":"9e397cd6a6f342065e6d59a0d15e23c5d5b2d99ae4a06377666083701a95e2aa"}
Sep 14 11:02:46.431613 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.431495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx" event={"ID":"922ac942-96fd-4206-b6c6-25c61035427e","Type":"ContainerStarted","Data":"9887ecb87f93e3ca1eb3a75835a9bb3376d5162db201aacf4415d94302bfec23"}
Sep 14 11:02:46.451489 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.451174    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-zrc4g" podStartSLOduration=1.255074002 podStartE2EDuration="2.451153675s" totalImagesPullingTime="1.196079673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:46.449222134 +0000 UTC m=+115.071775353" watchObservedRunningTime="2026-09-14 11:02:46.451153675 +0000 UTC m=+115.073706893"
Sep 14 11:02:46.456887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:46.456859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:47.433870 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.433835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:47.438245 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.438215    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca11ed89-1292-43c6-84fe-cb2035d1f52c" containerID="53c749c642a1e2045c740fab6c9c5c044e628a32ae756a5f11d0a9dc689cd3f2" exitCode=0
Sep 14 11:02:47.438412 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.438301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bn8j" event={"ID":"ca11ed89-1292-43c6-84fe-cb2035d1f52c","Type":"ContainerDied","Data":"53c749c642a1e2045c740fab6c9c5c044e628a32ae756a5f11d0a9dc689cd3f2"}
Sep 14 11:02:47.440946 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.440909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx" event={"ID":"922ac942-96fd-4206-b6c6-25c61035427e","Type":"ContainerStarted","Data":"542aaeefb1bd5275594750fb9ad6dd242cfe426eabd9e4f47965c2fef32040eb"}
Sep 14 11:02:47.511725 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.511670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-769cx" podStartSLOduration=2.103569097 podStartE2EDuration="3.511646882s" totalImagesPullingTime="1.408077785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:47.509424787 +0000 UTC m=+116.131978002" watchObservedRunningTime="2026-09-14 11:02:47.511646882 +0000 UTC m=+116.134200097"
Sep 14 11:02:47.524909 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.524805    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"]
Sep 14 11:02:47.529040 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.529021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.536313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.536288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 14 11:02:47.536451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.536353    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 14 11:02:47.536590 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.536553    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 14 11:02:47.536590 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.536582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s8x7f\""
Sep 14 11:02:47.536753 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.536614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 14 11:02:47.536824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.536798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 14 11:02:47.543327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.543308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-88vtvnua688jf\""
Sep 14 11:02:47.561144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.561115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:47.573407 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.573367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"]
Sep 14 11:02:47.619972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.619934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-tls\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.619972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.619981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-grpc-tls\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.620157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.620016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.620157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.620053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.620157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.620116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.620254 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.620161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6k8\" (UniqueName: \"kubernetes.io/projected/d11741f4-df64-483c-a3cf-16d014c71751-kube-api-access-kp6k8\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.620254 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.620192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.620337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.620261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11741f4-df64-483c-a3cf-16d014c71751-metrics-client-ca\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.721576 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-tls\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.721576 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-grpc-tls\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.721576 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.721865 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.721917 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.721964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6k8\" (UniqueName: \"kubernetes.io/projected/d11741f4-df64-483c-a3cf-16d014c71751-kube-api-access-kp6k8\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.722020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.721979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.722077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.722036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11741f4-df64-483c-a3cf-16d014c71751-metrics-client-ca\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.722828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.722802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d11741f4-df64-483c-a3cf-16d014c71751-metrics-client-ca\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.725014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.724966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.725112 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.725057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.725213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.725187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.725271 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.725254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-tls\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.725534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.725511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-grpc-tls\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.726587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.726566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d11741f4-df64-483c-a3cf-16d014c71751-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.733658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.733638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6k8\" (UniqueName: \"kubernetes.io/projected/d11741f4-df64-483c-a3cf-16d014c71751-kube-api-access-kp6k8\") pod \"thanos-querier-7d44f79b99-zz4j2\" (UID: \"d11741f4-df64-483c-a3cf-16d014c71751\") " pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.817607 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.817576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:47.860591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.860558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:47.912573 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:47.912380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:48.146016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:48.145992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"]
Sep 14 11:02:48.148110 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:48.148083    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11741f4_df64_483c_a3cf_16d014c71751.slice/crio-5a4b3d2df2462b991a6a657d94b883bd9cac9a1ce94e2fe5dd0fa7b7a0c48fec WatchSource:0}: Error finding container 5a4b3d2df2462b991a6a657d94b883bd9cac9a1ce94e2fe5dd0fa7b7a0c48fec: Status 404 returned error can't find the container with id 5a4b3d2df2462b991a6a657d94b883bd9cac9a1ce94e2fe5dd0fa7b7a0c48fec
Sep 14 11:02:48.447355 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:48.447272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bn8j" event={"ID":"ca11ed89-1292-43c6-84fe-cb2035d1f52c","Type":"ContainerStarted","Data":"dad75ffdb90180d8a934e1a13043193045cef9f543bd970b7d57a3b456226763"}
Sep 14 11:02:48.447355 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:48.447338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bn8j" event={"ID":"ca11ed89-1292-43c6-84fe-cb2035d1f52c","Type":"ContainerStarted","Data":"f2d63b8bcfcf90c93f53fe3e156481eb767e157be756a657ffda15247a44c1b6"}
Sep 14 11:02:48.448575 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:48.448550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"5a4b3d2df2462b991a6a657d94b883bd9cac9a1ce94e2fe5dd0fa7b7a0c48fec"}
Sep 14 11:02:48.471858 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:48.471810    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5bn8j" podStartSLOduration=2.593037844 podStartE2EDuration="4.471795603s" totalImagesPullingTime="1.812213689s" totalInitContainerRuntime="66.54407ms" isStatefulPod=true podCreationTimestamp="2026-09-14 11:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:48.470198217 +0000 UTC m=+117.092751431" watchObservedRunningTime="2026-09-14 11:02:48.471795603 +0000 UTC m=+117.094348857"
Sep 14 11:02:49.711074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:49.711039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:49.785107 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:49.785073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:50.870767 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:50.870628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:51.045566 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:51.045525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:51.466234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:51.466039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"fd9bed43e57d0c5973db21401fc549067853cf79e2ef7704b12b2709afe15d9a"}
Sep 14 11:02:51.466234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:51.466161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"1f6e6ddbf6e686a9b67143f64be04e4ed83105bb7db26fbcaaeead9d485ff037"}
Sep 14 11:02:51.466234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:51.466209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"851490d5a1d13acb85ca68b97a6be5ca2d200981c1b0775c8891d12328008ab1"}
Sep 14 11:02:52.474292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:52.474250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"1a614d8d91445d7c5729809c2f6472f94b65318be1bb05f0d9f5e2c45af14f7d"}
Sep 14 11:02:52.474707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:52.474340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"728eab8e21384baf63e6da09d763bd5edecc9cf8c8ef7a7cc76bf450da54394a"}
Sep 14 11:02:52.474707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:52.474369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" event={"ID":"d11741f4-df64-483c-a3cf-16d014c71751","Type":"ContainerStarted","Data":"32787bd1b6afe8ff91981659e48e7b82a3d5ea6e1136049e1a96634f924a038e"}
Sep 14 11:02:52.474707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:52.474423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:52.552153 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:52.552122    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5647bc96b4-bh85d"]
Sep 14 11:02:52.552694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:52.552645    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2" podStartSLOduration=2.451230802 podStartE2EDuration="5.552629337s" totalImagesPullingTime="3.101398535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:52.548744336 +0000 UTC m=+121.171297551" watchObservedRunningTime="2026-09-14 11:02:52.552629337 +0000 UTC m=+121.175182550"
Sep 14 11:02:53.073662 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:53.073630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:53.143650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:53.143621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:54.001740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.001697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:54.102632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.102597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:54.582474 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.582442    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-795dd96c76-w68g7"]
Sep 14 11:02:54.588319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.588298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.597652 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.597629    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 11:02:54.611319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.611289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795dd96c76-w68g7"]
Sep 14 11:02:54.688501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-oauth-serving-cert\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.688698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-console-config\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.688698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-service-ca\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.688698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-trusted-ca-bundle\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.688698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-serving-cert\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.688698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-oauth-config\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.688901 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.688736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6cc\" (UniqueName: \"kubernetes.io/projected/84c23112-d8b0-4c5b-a950-736f5625864c-kube-api-access-8s6cc\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-oauth-config\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6cc\" (UniqueName: \"kubernetes.io/projected/84c23112-d8b0-4c5b-a950-736f5625864c-kube-api-access-8s6cc\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-oauth-serving-cert\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-console-config\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-service-ca\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-trusted-ca-bundle\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.789857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.789847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-serving-cert\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.790657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.790624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-console-config\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.790788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.790685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-oauth-serving-cert\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.790788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.790686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-service-ca\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.790788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.790739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-trusted-ca-bundle\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.792144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.792126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-oauth-config\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.792379 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.792362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-serving-cert\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.798959 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.798937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6cc\" (UniqueName: \"kubernetes.io/projected/84c23112-d8b0-4c5b-a950-736f5625864c-kube-api-access-8s6cc\") pod \"console-795dd96c76-w68g7\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") " pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:54.898031 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:54.897998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:02:55.040932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.040899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:55.172064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.172034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795dd96c76-w68g7"]
Sep 14 11:02:55.174869 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:55.174841    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c23112_d8b0_4c5b_a950_736f5625864c.slice/crio-a04cef6f19f44770adad90c3f65f7f9e590cf839e88ce182dd2ed4f4eba8320c WatchSource:0}: Error finding container a04cef6f19f44770adad90c3f65f7f9e590cf839e88ce182dd2ed4f4eba8320c: Status 404 returned error can't find the container with id a04cef6f19f44770adad90c3f65f7f9e590cf839e88ce182dd2ed4f4eba8320c
Sep 14 11:02:55.216205 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.216175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:55.336796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.336765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:55.488295 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.488208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795dd96c76-w68g7" event={"ID":"84c23112-d8b0-4c5b-a950-736f5625864c","Type":"ContainerStarted","Data":"b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c"}
Sep 14 11:02:55.488295 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.488285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795dd96c76-w68g7" event={"ID":"84c23112-d8b0-4c5b-a950-736f5625864c","Type":"ContainerStarted","Data":"a04cef6f19f44770adad90c3f65f7f9e590cf839e88ce182dd2ed4f4eba8320c"}
Sep 14 11:02:55.672520 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.672484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:55.790386 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:55.790298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:56.806823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.806788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:56.815120 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.815065    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-795dd96c76-w68g7" podStartSLOduration=2.815044681 podStartE2EDuration="2.815044681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:55.546354618 +0000 UTC m=+124.168907832" watchObservedRunningTime="2026-09-14 11:02:56.815044681 +0000 UTC m=+125.437597896"
Sep 14 11:02:56.816055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.816029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-795dd96c76-w68g7"]
Sep 14 11:02:56.881079 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.881039    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65b9cf77f4-v9sjb"]
Sep 14 11:02:56.885086 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.885062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:56.910569 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.910541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65b9cf77f4-v9sjb"]
Sep 14 11:02:56.914656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:56.914629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:57.011171 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-oauth-config\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.011348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-config\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.011348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-service-ca\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.011348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-serving-cert\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.011348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-oauth-serving-cert\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.011646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-trusted-ca-bundle\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.011646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.011536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjpl\" (UniqueName: \"kubernetes.io/projected/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-kube-api-access-zmjpl\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.112984 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.112943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjpl\" (UniqueName: \"kubernetes.io/projected/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-kube-api-access-zmjpl\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.112984 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.112990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-oauth-config\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.113216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.113019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-config\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.113267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.113220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-service-ca\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.113337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.113263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-serving-cert\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.113337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.113291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-oauth-serving-cert\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.113484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.113349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-trusted-ca-bundle\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.114023 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.113997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-oauth-serving-cert\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.114140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.114020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-service-ca\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.114140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.114005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-config\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.114218 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.114158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-trusted-ca-bundle\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.115649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.115629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-oauth-config\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.116037 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.116020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-serving-cert\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.131513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.131456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjpl\" (UniqueName: \"kubernetes.io/projected/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-kube-api-access-zmjpl\") pod \"console-65b9cf77f4-v9sjb\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") " pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.194636 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.194600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:02:57.482875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.482821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65b9cf77f4-v9sjb"]
Sep 14 11:02:57.485058 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:02:57.485030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb82f4a9_b67d_4536_b8fe_bb74298e9a16.slice/crio-0c00991ec035d6ee3bdbdca35fa138f8ccacfec9faefb4c402c1a7554f13b626 WatchSource:0}: Error finding container 0c00991ec035d6ee3bdbdca35fa138f8ccacfec9faefb4c402c1a7554f13b626: Status 404 returned error can't find the container with id 0c00991ec035d6ee3bdbdca35fa138f8ccacfec9faefb4c402c1a7554f13b626
Sep 14 11:02:57.498744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:57.498714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b9cf77f4-v9sjb" event={"ID":"fb82f4a9-b67d-4536-b8fe-bb74298e9a16","Type":"ContainerStarted","Data":"0c00991ec035d6ee3bdbdca35fa138f8ccacfec9faefb4c402c1a7554f13b626"}
Sep 14 11:02:58.484766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:58.484739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d44f79b99-zz4j2"
Sep 14 11:02:58.503964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:58.503926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b9cf77f4-v9sjb" event={"ID":"fb82f4a9-b67d-4536-b8fe-bb74298e9a16","Type":"ContainerStarted","Data":"f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27"}
Sep 14 11:02:58.587656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:02:58.587605    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65b9cf77f4-v9sjb" podStartSLOduration=2.5875882629999998 podStartE2EDuration="2.587588263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:58.586225048 +0000 UTC m=+127.208778312" watchObservedRunningTime="2026-09-14 11:02:58.587588263 +0000 UTC m=+127.210141477"
Sep 14 11:03:01.755188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:01.755139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:03:01.757533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:01.757503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067aa408-be71-402b-aada-cd64b57c95f0-metrics-certs\") pod \"network-metrics-daemon-jplcp\" (UID: \"067aa408-be71-402b-aada-cd64b57c95f0\") " pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:03:02.025933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:02.025847    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npv4p\""
Sep 14 11:03:02.033772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:02.033751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jplcp"
Sep 14 11:03:02.306137 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:02.306052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jplcp"]
Sep 14 11:03:02.309370 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:03:02.309333    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067aa408_be71_402b_aada_cd64b57c95f0.slice/crio-58845161fe9d9acb9ab53e384c6d18135f5abb9f0fe33d2473cfca1a8e04b759 WatchSource:0}: Error finding container 58845161fe9d9acb9ab53e384c6d18135f5abb9f0fe33d2473cfca1a8e04b759: Status 404 returned error can't find the container with id 58845161fe9d9acb9ab53e384c6d18135f5abb9f0fe33d2473cfca1a8e04b759
Sep 14 11:03:02.521732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:02.521687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jplcp" event={"ID":"067aa408-be71-402b-aada-cd64b57c95f0","Type":"ContainerStarted","Data":"58845161fe9d9acb9ab53e384c6d18135f5abb9f0fe33d2473cfca1a8e04b759"}
Sep 14 11:03:03.164588 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:03.164555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:03.265852 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:03.265804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:04.519573 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:04.519542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:04.619265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:04.619242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:04.898654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:04.898611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:03:05.537783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:05.537732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jplcp" event={"ID":"067aa408-be71-402b-aada-cd64b57c95f0","Type":"ContainerStarted","Data":"236f641b5bb06b9977e0f82fade4ff26eaa84ee4ecee0dcf8154ad1b2ae3bd61"}
Sep 14 11:03:05.538186 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:05.537808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jplcp" event={"ID":"067aa408-be71-402b-aada-cd64b57c95f0","Type":"ContainerStarted","Data":"348493f764d4fa1ac8dfe4ab0086832e2a60599921abfc3a221e9cce9dd34b12"}
Sep 14 11:03:05.584087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:05.584030    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jplcp" podStartSLOduration=131.279635752 podStartE2EDuration="2m13.584013771s" totalImagesPullingTime="2.304378019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:05.583590943 +0000 UTC m=+134.206144171" watchObservedRunningTime="2026-09-14 11:03:05.584013771 +0000 UTC m=+134.206567029"
Sep 14 11:03:05.822950 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:05.822849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:05.922759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:05.922725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:07.195312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.195265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:03:07.195698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.195334    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:03:07.201505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.201474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:03:07.552916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.552834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:03:07.639755 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.639721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:07.705890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.705858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:07.806670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.806590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:07.910666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.910634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:07.981627 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:07.981574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:10.219644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:10.219608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:10.282547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:10.282514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:15.582733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:15.582702    2563 generic.go:356] "Generic (PLEG): container finished" podID="cbefa077-d861-42b7-be43-c4c3b0bdfaba" containerID="63c17f254aa874068fc099ed019b96062f280114290fd8331ade13411c88bfb2" exitCode=0
Sep 14 11:03:15.583108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:15.582740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hnnpq" event={"ID":"cbefa077-d861-42b7-be43-c4c3b0bdfaba","Type":"ContainerDied","Data":"63c17f254aa874068fc099ed019b96062f280114290fd8331ade13411c88bfb2"}
Sep 14 11:03:15.583108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:15.583091    2563 scope.go:117] "RemoveContainer" containerID="63c17f254aa874068fc099ed019b96062f280114290fd8331ade13411c88bfb2"
Sep 14 11:03:16.593777 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:16.593741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hnnpq" event={"ID":"cbefa077-d861-42b7-be43-c4c3b0bdfaba","Type":"ContainerStarted","Data":"8dfd72f970fe32f90e96d2eb6011e75af24dff0b68f560039dd143422f6bc9c7"}
Sep 14 11:03:17.571756 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.571692    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5647bc96b4-bh85d" podUID="3acc29cb-b262-4eaf-9c20-59d6249cf99f" containerName="console" containerID="cri-o://3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c" gracePeriod=15
Sep 14 11:03:17.818451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.818427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:03:17.899132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smjn2\" (UniqueName: \"kubernetes.io/projected/3acc29cb-b262-4eaf-9c20-59d6249cf99f-kube-api-access-smjn2\") pod \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") "
Sep 14 11:03:17.899370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-oauth-config\") pod \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") "
Sep 14 11:03:17.899370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-config\") pod \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") "
Sep 14 11:03:17.899370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899186    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-oauth-serving-cert\") pod \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") "
Sep 14 11:03:17.899370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-service-ca\") pod \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") "
Sep 14 11:03:17.899370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-serving-cert\") pod \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\" (UID: \"3acc29cb-b262-4eaf-9c20-59d6249cf99f\") "
Sep 14 11:03:17.899713 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-config" (OuterVolumeSpecName: "console-config") pod "3acc29cb-b262-4eaf-9c20-59d6249cf99f" (UID: "3acc29cb-b262-4eaf-9c20-59d6249cf99f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:17.899713 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3acc29cb-b262-4eaf-9c20-59d6249cf99f" (UID: "3acc29cb-b262-4eaf-9c20-59d6249cf99f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:17.900137 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.899703    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3acc29cb-b262-4eaf-9c20-59d6249cf99f" (UID: "3acc29cb-b262-4eaf-9c20-59d6249cf99f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:17.901618 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.901581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3acc29cb-b262-4eaf-9c20-59d6249cf99f" (UID: "3acc29cb-b262-4eaf-9c20-59d6249cf99f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:03:17.901732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.901636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3acc29cb-b262-4eaf-9c20-59d6249cf99f" (UID: "3acc29cb-b262-4eaf-9c20-59d6249cf99f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:03:17.901732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:17.901650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acc29cb-b262-4eaf-9c20-59d6249cf99f-kube-api-access-smjn2" (OuterVolumeSpecName: "kube-api-access-smjn2") pod "3acc29cb-b262-4eaf-9c20-59d6249cf99f" (UID: "3acc29cb-b262-4eaf-9c20-59d6249cf99f"). InnerVolumeSpecName "kube-api-access-smjn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:03:18.000747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.000704    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smjn2\" (UniqueName: \"kubernetes.io/projected/3acc29cb-b262-4eaf-9c20-59d6249cf99f-kube-api-access-smjn2\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:18.000747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.000742    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-oauth-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:18.001025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.000757    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:18.001025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.000770    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-oauth-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:18.001025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.000783    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3acc29cb-b262-4eaf-9c20-59d6249cf99f-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:18.001025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.000797    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3acc29cb-b262-4eaf-9c20-59d6249cf99f-console-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:18.602577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.602546    2563 generic.go:356] "Generic (PLEG): container finished" podID="3acc29cb-b262-4eaf-9c20-59d6249cf99f" containerID="3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c" exitCode=0
Sep 14 11:03:18.602751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.602591    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5647bc96b4-bh85d"
Sep 14 11:03:18.602751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.602633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647bc96b4-bh85d" event={"ID":"3acc29cb-b262-4eaf-9c20-59d6249cf99f","Type":"ContainerDied","Data":"3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c"}
Sep 14 11:03:18.602751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.602685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5647bc96b4-bh85d" event={"ID":"3acc29cb-b262-4eaf-9c20-59d6249cf99f","Type":"ContainerDied","Data":"5c5f8558199fd0dceb39eba78c1475833b6696ef4ab185ceccf2e8d714b406cd"}
Sep 14 11:03:18.602751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.602710    2563 scope.go:117] "RemoveContainer" containerID="3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c"
Sep 14 11:03:18.611050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.611033    2563 scope.go:117] "RemoveContainer" containerID="3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c"
Sep 14 11:03:18.611323 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:03:18.611300    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c\": container with ID starting with 3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c not found: ID does not exist" containerID="3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c"
Sep 14 11:03:18.611418 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.611337    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c"} err="failed to get container status \"3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c\": rpc error: code = NotFound desc = could not find container \"3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c\": container with ID starting with 3a7b98c12bcee6912bcfec2f1dfa5cb6bed5342c4e8509e46eab9f710aca050c not found: ID does not exist"
Sep 14 11:03:18.623608 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.623586    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5647bc96b4-bh85d"]
Sep 14 11:03:18.627180 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:18.627151    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5647bc96b4-bh85d"]
Sep 14 11:03:19.917519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:19.917484    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3acc29cb-b262-4eaf-9c20-59d6249cf99f" path="/var/lib/kubelet/pods/3acc29cb-b262-4eaf-9c20-59d6249cf99f/volumes"
Sep 14 11:03:22.517366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.517321    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-795dd96c76-w68g7" podUID="84c23112-d8b0-4c5b-a950-736f5625864c" containerName="console" containerID="cri-o://b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c" gracePeriod=15
Sep 14 11:03:22.762010 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.761982    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:03:22.837125 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-oauth-serving-cert\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837125 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-serving-cert\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837125 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-trusted-ca-bundle\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837125 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837129    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-service-ca\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-oauth-config\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s6cc\" (UniqueName: \"kubernetes.io/projected/84c23112-d8b0-4c5b-a950-736f5625864c-kube-api-access-8s6cc\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-console-config\") pod \"84c23112-d8b0-4c5b-a950-736f5625864c\" (UID: \"84c23112-d8b0-4c5b-a950-736f5625864c\") "
Sep 14 11:03:22.837707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-service-ca" (OuterVolumeSpecName: "service-ca") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:22.837707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:22.837707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-console-config" (OuterVolumeSpecName: "console-config") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:22.837960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.837929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:22.839365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.839335    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:03:22.839365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.839359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:03:22.839511 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.839437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c23112-d8b0-4c5b-a950-736f5625864c-kube-api-access-8s6cc" (OuterVolumeSpecName: "kube-api-access-8s6cc") pod "84c23112-d8b0-4c5b-a950-736f5625864c" (UID: "84c23112-d8b0-4c5b-a950-736f5625864c"). InnerVolumeSpecName "kube-api-access-8s6cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:03:22.938179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938136    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-oauth-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:22.938179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938170    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:22.938179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938181    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-trusted-ca-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:22.938179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938191    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:22.938485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938201    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84c23112-d8b0-4c5b-a950-736f5625864c-console-oauth-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:22.938485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938209    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6cc\" (UniqueName: \"kubernetes.io/projected/84c23112-d8b0-4c5b-a950-736f5625864c-kube-api-access-8s6cc\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:22.938485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:22.938219    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84c23112-d8b0-4c5b-a950-736f5625864c-console-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:03:23.625588 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.625558    2563 generic.go:356] "Generic (PLEG): container finished" podID="84c23112-d8b0-4c5b-a950-736f5625864c" containerID="b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c" exitCode=0
Sep 14 11:03:23.626030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.625605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795dd96c76-w68g7" event={"ID":"84c23112-d8b0-4c5b-a950-736f5625864c","Type":"ContainerDied","Data":"b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c"}
Sep 14 11:03:23.626030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.625618    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795dd96c76-w68g7"
Sep 14 11:03:23.626030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.625645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795dd96c76-w68g7" event={"ID":"84c23112-d8b0-4c5b-a950-736f5625864c","Type":"ContainerDied","Data":"a04cef6f19f44770adad90c3f65f7f9e590cf839e88ce182dd2ed4f4eba8320c"}
Sep 14 11:03:23.626030 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.625671    2563 scope.go:117] "RemoveContainer" containerID="b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c"
Sep 14 11:03:23.634902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.634886    2563 scope.go:117] "RemoveContainer" containerID="b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c"
Sep 14 11:03:23.635162 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:03:23.635143    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c\": container with ID starting with b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c not found: ID does not exist" containerID="b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c"
Sep 14 11:03:23.635225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.635170    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c"} err="failed to get container status \"b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c\": rpc error: code = NotFound desc = could not find container \"b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c\": container with ID starting with b05e7cdedffd25d565d4a6c89eaeed8696af6c1fcaf9bd19f7dafb02a2b31f0c not found: ID does not exist"
Sep 14 11:03:23.650428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.650405    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-795dd96c76-w68g7"]
Sep 14 11:03:23.653436 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.653417    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-795dd96c76-w68g7"]
Sep 14 11:03:23.918041 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:23.917960    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c23112-d8b0-4c5b-a950-736f5625864c" path="/var/lib/kubelet/pods/84c23112-d8b0-4c5b-a950-736f5625864c/volumes"
Sep 14 11:03:25.247632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:25.247591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:25.323730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:25.323697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:29.785543 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:03:29.785507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:00.662201 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:00.662161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:01.734126 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:01.734079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:12.613526 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:12.613492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:12.695534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:12.695490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:17.374657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.374620    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbdccf59c-qx9zw"]
Sep 14 11:04:17.375055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.374954    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acc29cb-b262-4eaf-9c20-59d6249cf99f" containerName="console"
Sep 14 11:04:17.375055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.374965    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acc29cb-b262-4eaf-9c20-59d6249cf99f" containerName="console"
Sep 14 11:04:17.375055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.374975    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c23112-d8b0-4c5b-a950-736f5625864c" containerName="console"
Sep 14 11:04:17.375055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.374980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c23112-d8b0-4c5b-a950-736f5625864c" containerName="console"
Sep 14 11:04:17.375055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.375033    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c23112-d8b0-4c5b-a950-736f5625864c" containerName="console"
Sep 14 11:04:17.375055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.375045    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acc29cb-b262-4eaf-9c20-59d6249cf99f" containerName="console"
Sep 14 11:04:17.377975 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.377951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384369 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqd4\" (UniqueName: \"kubernetes.io/projected/17da9208-bb8d-4d11-a042-8f4955cbe121-kube-api-access-vjqd4\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384517 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-oauth-config\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384517 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-console-config\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-oauth-serving-cert\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-serving-cert\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-service-ca\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.384929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.384695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-trusted-ca-bundle\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.396658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.396630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbdccf59c-qx9zw"]
Sep 14 11:04:17.450781 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.450742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:17.485644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.485608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-service-ca\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.485644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.485648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-trusted-ca-bundle\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.485889 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.485776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqd4\" (UniqueName: \"kubernetes.io/projected/17da9208-bb8d-4d11-a042-8f4955cbe121-kube-api-access-vjqd4\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.485889 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.485827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-oauth-config\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.485889 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.485849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-console-config\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.486075 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.485968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-oauth-serving-cert\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.486075 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.486018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-serving-cert\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.486335 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.486317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-service-ca\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.486510 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.486488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-console-config\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.486578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.486490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-trusted-ca-bundle\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.486883 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.486863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-oauth-serving-cert\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.488368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.488347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-oauth-config\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.488489 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.488473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-serving-cert\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.502490 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.502461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqd4\" (UniqueName: \"kubernetes.io/projected/17da9208-bb8d-4d11-a042-8f4955cbe121-kube-api-access-vjqd4\") pod \"console-cbdccf59c-qx9zw\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") " pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.517510 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.517483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:17.687223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.687123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:17.963270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:17.963092    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbdccf59c-qx9zw"]
Sep 14 11:04:17.965882 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:04:17.965856    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17da9208_bb8d_4d11_a042_8f4955cbe121.slice/crio-85cbb900d9899fad005410e02bf151133d71d0a559dbf2ebe9eb87c3f581bf76 WatchSource:0}: Error finding container 85cbb900d9899fad005410e02bf151133d71d0a559dbf2ebe9eb87c3f581bf76: Status 404 returned error can't find the container with id 85cbb900d9899fad005410e02bf151133d71d0a559dbf2ebe9eb87c3f581bf76
Sep 14 11:04:18.863409 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:18.863350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdccf59c-qx9zw" event={"ID":"17da9208-bb8d-4d11-a042-8f4955cbe121","Type":"ContainerStarted","Data":"8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41"}
Sep 14 11:04:18.863795 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:18.863429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdccf59c-qx9zw" event={"ID":"17da9208-bb8d-4d11-a042-8f4955cbe121","Type":"ContainerStarted","Data":"85cbb900d9899fad005410e02bf151133d71d0a559dbf2ebe9eb87c3f581bf76"}
Sep 14 11:04:18.884661 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:18.884551    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cbdccf59c-qx9zw" podStartSLOduration=1.8845306050000001 podStartE2EDuration="1.884530605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:18.884216998 +0000 UTC m=+207.506770225" watchObservedRunningTime="2026-09-14 11:04:18.884530605 +0000 UTC m=+207.507083820"
Sep 14 11:04:25.938988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:25.938952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:26.005056 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:26.005020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:27.688165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:27.688132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:27.688631 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:27.688214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:27.692995 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:27.692968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:27.903265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:27.903236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:04:27.962213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:27.962129    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65b9cf77f4-v9sjb"]
Sep 14 11:04:28.086687 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:28.086652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:28.187661 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:28.187624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:28.305813 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:28.305729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:45.319716 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:45.319680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:45.420300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:45.420260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:46.507957 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:46.507920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:46.608914 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:46.608884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:52.982510 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:52.982449    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65b9cf77f4-v9sjb" podUID="fb82f4a9-b67d-4536-b8fe-bb74298e9a16" containerName="console" containerID="cri-o://f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27" gracePeriod=15
Sep 14 11:04:53.221981 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.221952    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:04:53.389079 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-oauth-serving-cert\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389079 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-service-ca\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmjpl\" (UniqueName: \"kubernetes.io/projected/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-kube-api-access-zmjpl\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389173    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-trusted-ca-bundle\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389200    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-config\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-oauth-config\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-serving-cert\") pod \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\" (UID: \"fb82f4a9-b67d-4536-b8fe-bb74298e9a16\") "
Sep 14 11:04:53.389715 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:04:53.389840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:04:53.389840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:04:53.389840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.389835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-config" (OuterVolumeSpecName: "console-config") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:04:53.391517 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.391489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-kube-api-access-zmjpl" (OuterVolumeSpecName: "kube-api-access-zmjpl") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "kube-api-access-zmjpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:04:53.391757 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.391729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:04:53.391805 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.391751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb82f4a9-b67d-4536-b8fe-bb74298e9a16" (UID: "fb82f4a9-b67d-4536-b8fe-bb74298e9a16"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:04:53.409016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.408989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:53.490761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490726    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-oauth-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.490761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490753    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.490761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490764    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-oauth-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.490761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490772    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.491011 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490782    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjpl\" (UniqueName: \"kubernetes.io/projected/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-kube-api-access-zmjpl\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.491011 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490791    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-trusted-ca-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.491011 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.490799    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb82f4a9-b67d-4536-b8fe-bb74298e9a16-console-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:04:53.506720 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:53.506689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:04:54.010633 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.010593    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb82f4a9-b67d-4536-b8fe-bb74298e9a16" containerID="f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27" exitCode=0
Sep 14 11:04:54.011051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.010660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b9cf77f4-v9sjb"
Sep 14 11:04:54.011051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.010677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b9cf77f4-v9sjb" event={"ID":"fb82f4a9-b67d-4536-b8fe-bb74298e9a16","Type":"ContainerDied","Data":"f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27"}
Sep 14 11:04:54.011051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.010733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b9cf77f4-v9sjb" event={"ID":"fb82f4a9-b67d-4536-b8fe-bb74298e9a16","Type":"ContainerDied","Data":"0c00991ec035d6ee3bdbdca35fa138f8ccacfec9faefb4c402c1a7554f13b626"}
Sep 14 11:04:54.011051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.010752    2563 scope.go:117] "RemoveContainer" containerID="f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27"
Sep 14 11:04:54.019096 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.019078    2563 scope.go:117] "RemoveContainer" containerID="f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27"
Sep 14 11:04:54.019332 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:04:54.019315    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27\": container with ID starting with f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27 not found: ID does not exist" containerID="f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27"
Sep 14 11:04:54.019381 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.019362    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27"} err="failed to get container status \"f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27\": rpc error: code = NotFound desc = could not find container \"f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27\": container with ID starting with f5ea4652c6e900458c907baab38f9ab64d7a308e4dd479becbc666100e3a4a27 not found: ID does not exist"
Sep 14 11:04:54.030762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.030737    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65b9cf77f4-v9sjb"]
Sep 14 11:04:54.036316 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:54.036293    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65b9cf77f4-v9sjb"]
Sep 14 11:04:55.176433 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.176382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"]
Sep 14 11:04:55.176796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.176756    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb82f4a9-b67d-4536-b8fe-bb74298e9a16" containerName="console"
Sep 14 11:04:55.176796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.176770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb82f4a9-b67d-4536-b8fe-bb74298e9a16" containerName="console"
Sep 14 11:04:55.176881 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.176834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb82f4a9-b67d-4536-b8fe-bb74298e9a16" containerName="console"
Sep 14 11:04:55.181268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.181245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.183553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.183530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 11:04:55.183693 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.183639    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 11:04:55.184208 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.184190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sz8ts\""
Sep 14 11:04:55.192955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.192930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"]
Sep 14 11:04:55.307359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.307319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.307359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.307363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg67x\" (UniqueName: \"kubernetes.io/projected/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-kube-api-access-jg67x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.307660 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.307497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.408897 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.408859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.409051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.408898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg67x\" (UniqueName: \"kubernetes.io/projected/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-kube-api-access-jg67x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.409051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.408949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.409239 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.409222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.409278 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.409250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.417542 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.417512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg67x\" (UniqueName: \"kubernetes.io/projected/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-kube-api-access-jg67x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.490154 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.490052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:04:55.763403 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.763354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"]
Sep 14 11:04:55.765978 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:04:55.765944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821cc3c2_f6f2_4e53_9252_0f682f3b2f0f.slice/crio-7d75ddc4da9f867801a5dfd334b22f42fe8aae62face1cdd7eea6181d39888a6 WatchSource:0}: Error finding container 7d75ddc4da9f867801a5dfd334b22f42fe8aae62face1cdd7eea6181d39888a6: Status 404 returned error can't find the container with id 7d75ddc4da9f867801a5dfd334b22f42fe8aae62face1cdd7eea6181d39888a6
Sep 14 11:04:55.917687 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:55.917652    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb82f4a9-b67d-4536-b8fe-bb74298e9a16" path="/var/lib/kubelet/pods/fb82f4a9-b67d-4536-b8fe-bb74298e9a16/volumes"
Sep 14 11:04:56.020686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:04:56.020597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8" event={"ID":"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f","Type":"ContainerStarted","Data":"7d75ddc4da9f867801a5dfd334b22f42fe8aae62face1cdd7eea6181d39888a6"}
Sep 14 11:05:03.056313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:03.056278    2563 generic.go:356] "Generic (PLEG): container finished" podID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerID="e4b0781f328481484988c67b24684fe30ac76745b52d8afa9df2ebcdcb1497bb" exitCode=0
Sep 14 11:05:03.056710 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:03.056332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8" event={"ID":"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f","Type":"ContainerDied","Data":"e4b0781f328481484988c67b24684fe30ac76745b52d8afa9df2ebcdcb1497bb"}
Sep 14 11:05:06.070070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:06.070036    2563 generic.go:356] "Generic (PLEG): container finished" podID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerID="d361fdd6b2300a1fed0c17e5d9c09c4f4b128cd2975a79b95d5df74a92df5fa3" exitCode=0
Sep 14 11:05:06.070463 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:06.070117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8" event={"ID":"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f","Type":"ContainerDied","Data":"d361fdd6b2300a1fed0c17e5d9c09c4f4b128cd2975a79b95d5df74a92df5fa3"}
Sep 14 11:05:13.104623 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:13.104589    2563 generic.go:356] "Generic (PLEG): container finished" podID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerID="c14d304b9cd1de5aa7d7596b0a2d39ab831f4e9b04f08adf1572a382b9cfdc42" exitCode=0
Sep 14 11:05:13.104968 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:13.104666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8" event={"ID":"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f","Type":"ContainerDied","Data":"c14d304b9cd1de5aa7d7596b0a2d39ab831f4e9b04f08adf1572a382b9cfdc42"}
Sep 14 11:05:14.233890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.233865    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:05:14.275272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.275244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg67x\" (UniqueName: \"kubernetes.io/projected/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-kube-api-access-jg67x\") pod \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") "
Sep 14 11:05:14.275435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.275280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-util\") pod \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") "
Sep 14 11:05:14.275435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.275301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-bundle\") pod \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\" (UID: \"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f\") "
Sep 14 11:05:14.276004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.275977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-bundle" (OuterVolumeSpecName: "bundle") pod "821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" (UID: "821cc3c2-f6f2-4e53-9252-0f682f3b2f0f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:05:14.277706 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.277669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-kube-api-access-jg67x" (OuterVolumeSpecName: "kube-api-access-jg67x") pod "821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" (UID: "821cc3c2-f6f2-4e53-9252-0f682f3b2f0f"). InnerVolumeSpecName "kube-api-access-jg67x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:05:14.279490 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.279470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-util" (OuterVolumeSpecName: "util") pod "821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" (UID: "821cc3c2-f6f2-4e53-9252-0f682f3b2f0f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:05:14.376212 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.376126    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg67x\" (UniqueName: \"kubernetes.io/projected/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-kube-api-access-jg67x\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:14.376212 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.376159    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:14.376212 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:14.376168    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/821cc3c2-f6f2-4e53-9252-0f682f3b2f0f-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:15.114792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:15.114760    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8"
Sep 14 11:05:15.115001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:15.114794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l8wb8" event={"ID":"821cc3c2-f6f2-4e53-9252-0f682f3b2f0f","Type":"ContainerDied","Data":"7d75ddc4da9f867801a5dfd334b22f42fe8aae62face1cdd7eea6181d39888a6"}
Sep 14 11:05:15.115001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:15.114840    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d75ddc4da9f867801a5dfd334b22f42fe8aae62face1cdd7eea6181d39888a6"
Sep 14 11:05:15.277405 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:15.277368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:15.376990 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:15.376908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:16.318526 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:16.318488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:17.382300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:17.382262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:18.325855 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.325806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"]
Sep 14 11:05:18.326275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326252    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="extract"
Sep 14 11:05:18.326275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326277    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="extract"
Sep 14 11:05:18.326485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326291    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="util"
Sep 14 11:05:18.326485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326298    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="util"
Sep 14 11:05:18.326485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326326    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="pull"
Sep 14 11:05:18.326485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326334    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="pull"
Sep 14 11:05:18.326485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.326431    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="821cc3c2-f6f2-4e53-9252-0f682f3b2f0f" containerName="extract"
Sep 14 11:05:18.330593 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.330571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.332878 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.332850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:05:18.332995 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.332976    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 14 11:05:18.333043 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.332987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-spmd2\""
Sep 14 11:05:18.345180 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.345155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"]
Sep 14 11:05:18.407898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.407860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/619479b0-4cb1-4511-9064-b23323c6320c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-grzbk\" (UID: \"619479b0-4cb1-4511-9064-b23323c6320c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.408280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.407904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvp7s\" (UniqueName: \"kubernetes.io/projected/619479b0-4cb1-4511-9064-b23323c6320c-kube-api-access-fvp7s\") pod \"cert-manager-operator-controller-manager-55c444bf86-grzbk\" (UID: \"619479b0-4cb1-4511-9064-b23323c6320c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.509001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.508961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/619479b0-4cb1-4511-9064-b23323c6320c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-grzbk\" (UID: \"619479b0-4cb1-4511-9064-b23323c6320c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.509199 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.509010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvp7s\" (UniqueName: \"kubernetes.io/projected/619479b0-4cb1-4511-9064-b23323c6320c-kube-api-access-fvp7s\") pod \"cert-manager-operator-controller-manager-55c444bf86-grzbk\" (UID: \"619479b0-4cb1-4511-9064-b23323c6320c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.509336 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.509311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/619479b0-4cb1-4511-9064-b23323c6320c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-grzbk\" (UID: \"619479b0-4cb1-4511-9064-b23323c6320c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.518566 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.518543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvp7s\" (UniqueName: \"kubernetes.io/projected/619479b0-4cb1-4511-9064-b23323c6320c-kube-api-access-fvp7s\") pod \"cert-manager-operator-controller-manager-55c444bf86-grzbk\" (UID: \"619479b0-4cb1-4511-9064-b23323c6320c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.639464 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.639417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"
Sep 14 11:05:18.951560 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:18.951534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk"]
Sep 14 11:05:18.954407 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:18.954361    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619479b0_4cb1_4511_9064_b23323c6320c.slice/crio-c0bfbc3a0d8b9044c9951b01aeda1bf2a53b9f394986d41a3e5e145bbeb541f5 WatchSource:0}: Error finding container c0bfbc3a0d8b9044c9951b01aeda1bf2a53b9f394986d41a3e5e145bbeb541f5: Status 404 returned error can't find the container with id c0bfbc3a0d8b9044c9951b01aeda1bf2a53b9f394986d41a3e5e145bbeb541f5
Sep 14 11:05:19.133304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:19.133264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk" event={"ID":"619479b0-4cb1-4511-9064-b23323c6320c","Type":"ContainerStarted","Data":"c0bfbc3a0d8b9044c9951b01aeda1bf2a53b9f394986d41a3e5e145bbeb541f5"}
Sep 14 11:05:20.145110 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:20.145077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:20.971620 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:20.971578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:21.144867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:21.144832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk" event={"ID":"619479b0-4cb1-4511-9064-b23323c6320c","Type":"ContainerStarted","Data":"9b657c2f30b76769d2e0ab89818070becb298820ef138a6a4c445bf0c991ebb0"}
Sep 14 11:05:21.172547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:21.172488    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-grzbk" podStartSLOduration=1.067846182 podStartE2EDuration="3.172469821s" totalImagesPullingTime="2.104623639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:21.171121379 +0000 UTC m=+269.793674605" watchObservedRunningTime="2026-09-14 11:05:21.172469821 +0000 UTC m=+269.795023035"
Sep 14 11:05:21.603298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:21.603269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:21.695441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:21.695403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:22.008061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.007955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:22.122544 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.122499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:22.334101 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.334022    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"]
Sep 14 11:05:22.337515 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.337497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.341587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.341562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sz8ts\""
Sep 14 11:05:22.341741 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.341723    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 11:05:22.342172 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.342156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 11:05:22.355443 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.355419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"]
Sep 14 11:05:22.442266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.442229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.442266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.442267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.442519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.442298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wcsm\" (UniqueName: \"kubernetes.io/projected/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-kube-api-access-5wcsm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.542694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.542657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.542694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.542697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.542908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.542730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wcsm\" (UniqueName: \"kubernetes.io/projected/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-kube-api-access-5wcsm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.543152 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.543131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.543196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.543133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.562921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.562893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wcsm\" (UniqueName: \"kubernetes.io/projected/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-kube-api-access-5wcsm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.646984 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.646947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:22.939883 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.939667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"]
Sep 14 11:05:22.942772 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:22.942738    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ae828e_d9c7_423a_a3bc_7b0dc8098213.slice/crio-0004d61abf5e3c63bae4641a762cd98d6b39e03929ea8c5de4c331b95352425c WatchSource:0}: Error finding container 0004d61abf5e3c63bae4641a762cd98d6b39e03929ea8c5de4c331b95352425c: Status 404 returned error can't find the container with id 0004d61abf5e3c63bae4641a762cd98d6b39e03929ea8c5de4c331b95352425c
Sep 14 11:05:22.985330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.985304    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jwhjg"]
Sep 14 11:05:22.989653 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.989623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:22.993265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.993241    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 14 11:05:22.993375 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.993286    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 14 11:05:22.993375 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:22.993290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fk87v\""
Sep 14 11:05:23.003246 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.003218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jwhjg"]
Sep 14 11:05:23.046171 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.046138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a43d1e-a337-4e7d-a540-0576fabc6739-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jwhjg\" (UID: \"69a43d1e-a337-4e7d-a540-0576fabc6739\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.046313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.046184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhq4t\" (UniqueName: \"kubernetes.io/projected/69a43d1e-a337-4e7d-a540-0576fabc6739-kube-api-access-vhq4t\") pod \"cert-manager-webhook-6486ffd997-jwhjg\" (UID: \"69a43d1e-a337-4e7d-a540-0576fabc6739\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.146621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.146583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a43d1e-a337-4e7d-a540-0576fabc6739-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jwhjg\" (UID: \"69a43d1e-a337-4e7d-a540-0576fabc6739\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.146778 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.146640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhq4t\" (UniqueName: \"kubernetes.io/projected/69a43d1e-a337-4e7d-a540-0576fabc6739-kube-api-access-vhq4t\") pod \"cert-manager-webhook-6486ffd997-jwhjg\" (UID: \"69a43d1e-a337-4e7d-a540-0576fabc6739\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.155122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.155094    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerID="9e0f504deda95a70634d73bbe33b15d17c80f8d1858b9547582cc71783856112" exitCode=0
Sep 14 11:05:23.155269 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.155175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf" event={"ID":"a5ae828e-d9c7-423a-a3bc-7b0dc8098213","Type":"ContainerDied","Data":"9e0f504deda95a70634d73bbe33b15d17c80f8d1858b9547582cc71783856112"}
Sep 14 11:05:23.155269 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.155220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf" event={"ID":"a5ae828e-d9c7-423a-a3bc-7b0dc8098213","Type":"ContainerStarted","Data":"0004d61abf5e3c63bae4641a762cd98d6b39e03929ea8c5de4c331b95352425c"}
Sep 14 11:05:23.161727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.161699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a43d1e-a337-4e7d-a540-0576fabc6739-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jwhjg\" (UID: \"69a43d1e-a337-4e7d-a540-0576fabc6739\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.161824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.161801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhq4t\" (UniqueName: \"kubernetes.io/projected/69a43d1e-a337-4e7d-a540-0576fabc6739-kube-api-access-vhq4t\") pod \"cert-manager-webhook-6486ffd997-jwhjg\" (UID: \"69a43d1e-a337-4e7d-a540-0576fabc6739\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.307842 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.307757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:23.582531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:23.582504    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jwhjg"]
Sep 14 11:05:23.585076 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:23.585043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a43d1e_a337_4e7d_a540_0576fabc6739.slice/crio-a482d31517dc901ed5cf04f3e10c74a21549dbc6a85e1d7ac0ba242035237262 WatchSource:0}: Error finding container a482d31517dc901ed5cf04f3e10c74a21549dbc6a85e1d7ac0ba242035237262: Status 404 returned error can't find the container with id a482d31517dc901ed5cf04f3e10c74a21549dbc6a85e1d7ac0ba242035237262
Sep 14 11:05:24.161921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:24.161882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg" event={"ID":"69a43d1e-a337-4e7d-a540-0576fabc6739","Type":"ContainerStarted","Data":"a482d31517dc901ed5cf04f3e10c74a21549dbc6a85e1d7ac0ba242035237262"}
Sep 14 11:05:24.415458 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:24.415355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:24.626248 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:24.626215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:25.174799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:25.174701    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerID="0cbef2c68e95bb500c3d25e3228967c765e237cf25825be2a9ddf4393f12c402" exitCode=0
Sep 14 11:05:25.174799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:25.174774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf" event={"ID":"a5ae828e-d9c7-423a-a3bc-7b0dc8098213","Type":"ContainerDied","Data":"0cbef2c68e95bb500c3d25e3228967c765e237cf25825be2a9ddf4393f12c402"}
Sep 14 11:05:26.180680 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:26.180650    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerID="2766e3f36a5390fde3f8856c56359f069f1f2ba04fdad7cf2de7b450afc0836a" exitCode=0
Sep 14 11:05:26.181122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:26.180738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf" event={"ID":"a5ae828e-d9c7-423a-a3bc-7b0dc8098213","Type":"ContainerDied","Data":"2766e3f36a5390fde3f8856c56359f069f1f2ba04fdad7cf2de7b450afc0836a"}
Sep 14 11:05:26.910734 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:26.910700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:27.013129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.013096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:27.324994 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.324967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:27.388102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.388066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wcsm\" (UniqueName: \"kubernetes.io/projected/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-kube-api-access-5wcsm\") pod \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") "
Sep 14 11:05:27.388305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.388110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-util\") pod \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") "
Sep 14 11:05:27.388305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.388147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-bundle\") pod \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\" (UID: \"a5ae828e-d9c7-423a-a3bc-7b0dc8098213\") "
Sep 14 11:05:27.388551 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.388521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-bundle" (OuterVolumeSpecName: "bundle") pod "a5ae828e-d9c7-423a-a3bc-7b0dc8098213" (UID: "a5ae828e-d9c7-423a-a3bc-7b0dc8098213"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:05:27.390289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.390262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-kube-api-access-5wcsm" (OuterVolumeSpecName: "kube-api-access-5wcsm") pod "a5ae828e-d9c7-423a-a3bc-7b0dc8098213" (UID: "a5ae828e-d9c7-423a-a3bc-7b0dc8098213"). InnerVolumeSpecName "kube-api-access-5wcsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:05:27.393622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.393587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-util" (OuterVolumeSpecName: "util") pod "a5ae828e-d9c7-423a-a3bc-7b0dc8098213" (UID: "a5ae828e-d9c7-423a-a3bc-7b0dc8098213"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:05:27.488930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.488842    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:27.488930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.488877    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wcsm\" (UniqueName: \"kubernetes.io/projected/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-kube-api-access-5wcsm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:27.488930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:27.488886    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5ae828e-d9c7-423a-a3bc-7b0dc8098213-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:28.193599 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:28.193499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg" event={"ID":"69a43d1e-a337-4e7d-a540-0576fabc6739","Type":"ContainerStarted","Data":"d1dc079d1ad2f4d68e36061929fd0fda567f702933d9cbde0f21a0cee4cac82d"}
Sep 14 11:05:28.193599 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:28.193563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:28.195450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:28.195422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf" event={"ID":"a5ae828e-d9c7-423a-a3bc-7b0dc8098213","Type":"ContainerDied","Data":"0004d61abf5e3c63bae4641a762cd98d6b39e03929ea8c5de4c331b95352425c"}
Sep 14 11:05:28.195450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:28.195451    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnr7jf"
Sep 14 11:05:28.195594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:28.195467    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0004d61abf5e3c63bae4641a762cd98d6b39e03929ea8c5de4c331b95352425c"
Sep 14 11:05:28.211586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:28.211527    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg" podStartSLOduration=2.676521809 podStartE2EDuration="6.211508401s" totalImagesPullingTime="3.534986592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:28.210467806 +0000 UTC m=+276.833021019" watchObservedRunningTime="2026-09-14 11:05:28.211508401 +0000 UTC m=+276.834061616"
Sep 14 11:05:30.155284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155248    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"]
Sep 14 11:05:30.155698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155611    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="pull"
Sep 14 11:05:30.155698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="pull"
Sep 14 11:05:30.155698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155634    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="extract"
Sep 14 11:05:30.155698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155639    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="extract"
Sep 14 11:05:30.155698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155649    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="util"
Sep 14 11:05:30.155698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155655    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="util"
Sep 14 11:05:30.155881 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.155709    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ae828e-d9c7-423a-a3bc-7b0dc8098213" containerName="extract"
Sep 14 11:05:30.158801 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.158785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.161518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.161497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fqgmg\""
Sep 14 11:05:30.171806 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.171782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"]
Sep 14 11:05:30.212929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.212900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b5353b4-dbe2-4b82-aebb-857d63347802-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-96zpb\" (UID: \"8b5353b4-dbe2-4b82-aebb-857d63347802\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.212929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.212932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plk8h\" (UniqueName: \"kubernetes.io/projected/8b5353b4-dbe2-4b82-aebb-857d63347802-kube-api-access-plk8h\") pod \"cert-manager-cainjector-559cdd8c87-96zpb\" (UID: \"8b5353b4-dbe2-4b82-aebb-857d63347802\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.313457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.313416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b5353b4-dbe2-4b82-aebb-857d63347802-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-96zpb\" (UID: \"8b5353b4-dbe2-4b82-aebb-857d63347802\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.313650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.313464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plk8h\" (UniqueName: \"kubernetes.io/projected/8b5353b4-dbe2-4b82-aebb-857d63347802-kube-api-access-plk8h\") pod \"cert-manager-cainjector-559cdd8c87-96zpb\" (UID: \"8b5353b4-dbe2-4b82-aebb-857d63347802\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.322999 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.322969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b5353b4-dbe2-4b82-aebb-857d63347802-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-96zpb\" (UID: \"8b5353b4-dbe2-4b82-aebb-857d63347802\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.323128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.323053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plk8h\" (UniqueName: \"kubernetes.io/projected/8b5353b4-dbe2-4b82-aebb-857d63347802-kube-api-access-plk8h\") pod \"cert-manager-cainjector-559cdd8c87-96zpb\" (UID: \"8b5353b4-dbe2-4b82-aebb-857d63347802\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.468015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.467924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"
Sep 14 11:05:30.745447 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:30.745417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-96zpb"]
Sep 14 11:05:30.747366 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:30.747334    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b5353b4_dbe2_4b82_aebb_857d63347802.slice/crio-1aacd33cb6a6b8d7848d5f245a60a21da07ea137c00e15917b47578bf4c70752 WatchSource:0}: Error finding container 1aacd33cb6a6b8d7848d5f245a60a21da07ea137c00e15917b47578bf4c70752: Status 404 returned error can't find the container with id 1aacd33cb6a6b8d7848d5f245a60a21da07ea137c00e15917b47578bf4c70752
Sep 14 11:05:31.215940 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:31.215904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb" event={"ID":"8b5353b4-dbe2-4b82-aebb-857d63347802","Type":"ContainerStarted","Data":"941ed2ec841cffb484d888ccac689a9cce7603712df7c4b84ea7b922953b4e00"}
Sep 14 11:05:31.216326 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:31.215949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb" event={"ID":"8b5353b4-dbe2-4b82-aebb-857d63347802","Type":"ContainerStarted","Data":"1aacd33cb6a6b8d7848d5f245a60a21da07ea137c00e15917b47578bf4c70752"}
Sep 14 11:05:31.234852 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:31.234785    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-96zpb" podStartSLOduration=1.234765334 podStartE2EDuration="1.234765334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:31.233034201 +0000 UTC m=+279.855587415" watchObservedRunningTime="2026-09-14 11:05:31.234765334 +0000 UTC m=+279.857318549"
Sep 14 11:05:32.315191 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:32.315130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:32.416249 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:32.416213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:32.584603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:32.584516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:32.931974 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:32.931939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:33.049121 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:33.049085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:34.202559 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:34.202527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jwhjg"
Sep 14 11:05:39.210794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:39.210761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:39.306642 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:39.306609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:41.860418 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:41.860358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:42.630667 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.630633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6cdms"]
Sep 14 11:05:42.634052 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.634034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.636363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.636337    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-brf2h\""
Sep 14 11:05:42.643507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.643479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6cdms"]
Sep 14 11:05:42.728623 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.728578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fdd25f3-a496-4ec7-98b0-1e138c39cd0e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6cdms\" (UID: \"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e\") " pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.728818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.728633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nw48\" (UniqueName: \"kubernetes.io/projected/7fdd25f3-a496-4ec7-98b0-1e138c39cd0e-kube-api-access-4nw48\") pod \"cert-manager-55b4d954d5-6cdms\" (UID: \"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e\") " pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.829677 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.829637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fdd25f3-a496-4ec7-98b0-1e138c39cd0e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6cdms\" (UID: \"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e\") " pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.829854 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.829687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nw48\" (UniqueName: \"kubernetes.io/projected/7fdd25f3-a496-4ec7-98b0-1e138c39cd0e-kube-api-access-4nw48\") pod \"cert-manager-55b4d954d5-6cdms\" (UID: \"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e\") " pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.838624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.838599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fdd25f3-a496-4ec7-98b0-1e138c39cd0e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6cdms\" (UID: \"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e\") " pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.838754 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.838736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nw48\" (UniqueName: \"kubernetes.io/projected/7fdd25f3-a496-4ec7-98b0-1e138c39cd0e-kube-api-access-4nw48\") pod \"cert-manager-55b4d954d5-6cdms\" (UID: \"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e\") " pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:42.943929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:42.943832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6cdms"
Sep 14 11:05:43.215331 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.215304    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6cdms"]
Sep 14 11:05:43.217875 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:43.217845    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fdd25f3_a496_4ec7_98b0_1e138c39cd0e.slice/crio-d745a8e2bcbd88525ba0abae1a68935619d4c3cb592bf69d0678c47e4980d7d0 WatchSource:0}: Error finding container d745a8e2bcbd88525ba0abae1a68935619d4c3cb592bf69d0678c47e4980d7d0: Status 404 returned error can't find the container with id d745a8e2bcbd88525ba0abae1a68935619d4c3cb592bf69d0678c47e4980d7d0
Sep 14 11:05:43.273128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.273097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6cdms" event={"ID":"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e","Type":"ContainerStarted","Data":"d745a8e2bcbd88525ba0abae1a68935619d4c3cb592bf69d0678c47e4980d7d0"}
Sep 14 11:05:43.728128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.728089    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"]
Sep 14 11:05:43.751407 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.751362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"]
Sep 14 11:05:43.751586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.751482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.754196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.754173    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sz8ts\""
Sep 14 11:05:43.754339 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.754198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 11:05:43.754339 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.754267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 11:05:43.839057 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.839023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.839220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.839068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.839220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.839090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbg8\" (UniqueName: \"kubernetes.io/projected/7d910183-073e-4620-a965-07f941601592-kube-api-access-sdbg8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.939555 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.939522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.939747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.939575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.939747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.939602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbg8\" (UniqueName: \"kubernetes.io/projected/7d910183-073e-4620-a965-07f941601592-kube-api-access-sdbg8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.939986 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.939960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.940049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.939969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:43.951622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:43.951592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbg8\" (UniqueName: \"kubernetes.io/projected/7d910183-073e-4620-a965-07f941601592-kube-api-access-sdbg8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p\" (UID: \"7d910183-073e-4620-a965-07f941601592\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:44.060649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:44.060564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:44.279363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:44.279326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6cdms" event={"ID":"7fdd25f3-a496-4ec7-98b0-1e138c39cd0e","Type":"ContainerStarted","Data":"51166c067367131aa7026325bbf5751f5392c9d7c411b2943a40df0b8a724715"}
Sep 14 11:05:44.380195 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:44.380130    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6cdms" podStartSLOduration=2.38010862 podStartE2EDuration="2.38010862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:44.300279762 +0000 UTC m=+292.922832976" watchObservedRunningTime="2026-09-14 11:05:44.38010862 +0000 UTC m=+293.002661835"
Sep 14 11:05:44.380876 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:44.380858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"]
Sep 14 11:05:44.383424 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:44.383377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d910183_073e_4620_a965_07f941601592.slice/crio-2a3c8e39a5b8b517c7e43e71338f9dbffe234f7714f31c4be709bafaebddd3cf WatchSource:0}: Error finding container 2a3c8e39a5b8b517c7e43e71338f9dbffe234f7714f31c4be709bafaebddd3cf: Status 404 returned error can't find the container with id 2a3c8e39a5b8b517c7e43e71338f9dbffe234f7714f31c4be709bafaebddd3cf
Sep 14 11:05:45.286117 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:45.286083    2563 generic.go:356] "Generic (PLEG): container finished" podID="7d910183-073e-4620-a965-07f941601592" containerID="3681917a0a3c8be2ca7cdf0b536ba944a533db8fb4f26f53410e40848a68529f" exitCode=0
Sep 14 11:05:45.286558 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:45.286173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p" event={"ID":"7d910183-073e-4620-a965-07f941601592","Type":"ContainerDied","Data":"3681917a0a3c8be2ca7cdf0b536ba944a533db8fb4f26f53410e40848a68529f"}
Sep 14 11:05:45.286558 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:45.286231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p" event={"ID":"7d910183-073e-4620-a965-07f941601592","Type":"ContainerStarted","Data":"2a3c8e39a5b8b517c7e43e71338f9dbffe234f7714f31c4be709bafaebddd3cf"}
Sep 14 11:05:45.942944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:45.942910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:46.007905 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:46.007872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:47.298451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:47.298412    2563 generic.go:356] "Generic (PLEG): container finished" podID="7d910183-073e-4620-a965-07f941601592" containerID="7090204ef191c11f9f15a07100261e8b5a922544137a2103882a6248d8afed5a" exitCode=0
Sep 14 11:05:47.298924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:47.298533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p" event={"ID":"7d910183-073e-4620-a965-07f941601592","Type":"ContainerDied","Data":"7090204ef191c11f9f15a07100261e8b5a922544137a2103882a6248d8afed5a"}
Sep 14 11:05:48.305654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:48.305619    2563 generic.go:356] "Generic (PLEG): container finished" podID="7d910183-073e-4620-a965-07f941601592" containerID="3df1260e310ad03eaeede30084bf0bc4db2d6d19e5249ce4bb3777980e353ea7" exitCode=0
Sep 14 11:05:48.306038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:48.305696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p" event={"ID":"7d910183-073e-4620-a965-07f941601592","Type":"ContainerDied","Data":"3df1260e310ad03eaeede30084bf0bc4db2d6d19e5249ce4bb3777980e353ea7"}
Sep 14 11:05:49.440661 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.440631    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:49.586964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.586861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdbg8\" (UniqueName: \"kubernetes.io/projected/7d910183-073e-4620-a965-07f941601592-kube-api-access-sdbg8\") pod \"7d910183-073e-4620-a965-07f941601592\" (UID: \"7d910183-073e-4620-a965-07f941601592\") "
Sep 14 11:05:49.586964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.586915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-util\") pod \"7d910183-073e-4620-a965-07f941601592\" (UID: \"7d910183-073e-4620-a965-07f941601592\") "
Sep 14 11:05:49.586964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.586932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-bundle\") pod \"7d910183-073e-4620-a965-07f941601592\" (UID: \"7d910183-073e-4620-a965-07f941601592\") "
Sep 14 11:05:49.587703 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.587663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-bundle" (OuterVolumeSpecName: "bundle") pod "7d910183-073e-4620-a965-07f941601592" (UID: "7d910183-073e-4620-a965-07f941601592"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:05:49.589035 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.589001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d910183-073e-4620-a965-07f941601592-kube-api-access-sdbg8" (OuterVolumeSpecName: "kube-api-access-sdbg8") pod "7d910183-073e-4620-a965-07f941601592" (UID: "7d910183-073e-4620-a965-07f941601592"). InnerVolumeSpecName "kube-api-access-sdbg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:05:49.592480 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.592459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-util" (OuterVolumeSpecName: "util") pod "7d910183-073e-4620-a965-07f941601592" (UID: "7d910183-073e-4620-a965-07f941601592"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:05:49.688309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.688274    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbg8\" (UniqueName: \"kubernetes.io/projected/7d910183-073e-4620-a965-07f941601592-kube-api-access-sdbg8\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:49.688309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.688304    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:49.688309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:49.688316    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d910183-073e-4620-a965-07f941601592-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:05:50.317192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:50.317106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p" event={"ID":"7d910183-073e-4620-a965-07f941601592","Type":"ContainerDied","Data":"2a3c8e39a5b8b517c7e43e71338f9dbffe234f7714f31c4be709bafaebddd3cf"}
Sep 14 11:05:50.317192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:50.317131    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk8t5p"
Sep 14 11:05:50.317192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:50.317161    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3c8e39a5b8b517c7e43e71338f9dbffe234f7714f31c4be709bafaebddd3cf"
Sep 14 11:05:51.802737 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:51.802712    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:05:51.804104 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:51.804083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:05:51.816054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:51.816032    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 11:05:52.712168 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:52.712111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:58.497181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"]
Sep 14 11:05:58.497578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497495    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d910183-073e-4620-a965-07f941601592" containerName="extract"
Sep 14 11:05:58.497578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497508    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d910183-073e-4620-a965-07f941601592" containerName="extract"
Sep 14 11:05:58.497578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497517    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d910183-073e-4620-a965-07f941601592" containerName="util"
Sep 14 11:05:58.497578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d910183-073e-4620-a965-07f941601592" containerName="util"
Sep 14 11:05:58.497578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497527    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d910183-073e-4620-a965-07f941601592" containerName="pull"
Sep 14 11:05:58.497578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d910183-073e-4620-a965-07f941601592" containerName="pull"
Sep 14 11:05:58.497764 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.497605    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d910183-073e-4620-a965-07f941601592" containerName="extract"
Sep 14 11:05:58.541166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.541139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"]
Sep 14 11:05:58.541323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.541254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.546054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.546029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 11:05:58.547376 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.547216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sz8ts\""
Sep 14 11:05:58.550084 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.550065    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 11:05:58.663463 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.663424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.663734 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.663713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmgm\" (UniqueName: \"kubernetes.io/projected/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-kube-api-access-wjmgm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.663850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.663830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.764952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.764859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.764952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.764915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmgm\" (UniqueName: \"kubernetes.io/projected/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-kube-api-access-wjmgm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.764952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.764934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.765284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.765265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.765322 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.765289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.797798 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.797755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmgm\" (UniqueName: \"kubernetes.io/projected/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-kube-api-access-wjmgm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:58.855484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:58.855451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:05:59.172520 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.172488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"]
Sep 14 11:05:59.174701 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:05:59.174667    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ca7755_a645_4ff1_ac7b_1f5f33cabe5a.slice/crio-1f488b89e2f12c5051439dba012ed8c4e7505fc1200881346bfa4ffe47e308b7 WatchSource:0}: Error finding container 1f488b89e2f12c5051439dba012ed8c4e7505fc1200881346bfa4ffe47e308b7: Status 404 returned error can't find the container with id 1f488b89e2f12c5051439dba012ed8c4e7505fc1200881346bfa4ffe47e308b7
Sep 14 11:05:59.220731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.220700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:59.363515 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.363482    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerID="be01575c74eaedbd3bf827da1f3499aa22e4f830592cef30521a8311e37b413c" exitCode=0
Sep 14 11:05:59.363683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.363542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52" event={"ID":"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a","Type":"ContainerDied","Data":"be01575c74eaedbd3bf827da1f3499aa22e4f830592cef30521a8311e37b413c"}
Sep 14 11:05:59.363683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.363576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52" event={"ID":"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a","Type":"ContainerStarted","Data":"1f488b89e2f12c5051439dba012ed8c4e7505fc1200881346bfa4ffe47e308b7"}
Sep 14 11:05:59.364037 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.364023    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:05:59.589482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.589375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:05:59.792511 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:05:59.792468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.162736 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.162701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.261704 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.261664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.338261 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.338231    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"]
Sep 14 11:06:00.341730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.341714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.347260 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.347234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 11:06:00.347647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.347625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 11:06:00.347749 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.347652    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 11:06:00.347812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.347765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 11:06:00.350688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.350190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wrjmp\""
Sep 14 11:06:00.364222 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.364196    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"]
Sep 14 11:06:00.372651 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.372623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52" event={"ID":"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a","Type":"ContainerStarted","Data":"1bc9daa018f941ca778a5cc6dbfe69020c40dbc3015ce33ba25c503880152cb4"}
Sep 14 11:06:00.373946 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.373891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj88v\" (UniqueName: \"kubernetes.io/projected/1bb88491-f609-4a24-81f5-d7b0809d79ba-kube-api-access-tj88v\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.373946 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.373928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb88491-f609-4a24-81f5-d7b0809d79ba-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.374047 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.373947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bb88491-f609-4a24-81f5-d7b0809d79ba-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.374095 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.374049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bb88491-f609-4a24-81f5-d7b0809d79ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.374137 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.374123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bb88491-f609-4a24-81f5-d7b0809d79ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.474570 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.474536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bb88491-f609-4a24-81f5-d7b0809d79ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.474748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.474603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bb88491-f609-4a24-81f5-d7b0809d79ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.474748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.474638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj88v\" (UniqueName: \"kubernetes.io/projected/1bb88491-f609-4a24-81f5-d7b0809d79ba-kube-api-access-tj88v\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.474748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.474673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb88491-f609-4a24-81f5-d7b0809d79ba-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.474748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.474693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bb88491-f609-4a24-81f5-d7b0809d79ba-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.475161 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.475126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bb88491-f609-4a24-81f5-d7b0809d79ba-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.477330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.477300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb88491-f609-4a24-81f5-d7b0809d79ba-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.477482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.477364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bb88491-f609-4a24-81f5-d7b0809d79ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.477675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.477656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bb88491-f609-4a24-81f5-d7b0809d79ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.502045 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.502018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj88v\" (UniqueName: \"kubernetes.io/projected/1bb88491-f609-4a24-81f5-d7b0809d79ba-kube-api-access-tj88v\") pod \"opendatahub-operator-controller-manager-6f85645c65-8xv8p\" (UID: \"1bb88491-f609-4a24-81f5-d7b0809d79ba\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.561155 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.561125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.652318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.652287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:00.652796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.652449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.751254 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.751216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.856736 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.856696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.932340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.932308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:00.945925 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:00.945900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"]
Sep 14 11:06:00.947705 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:06:00.947675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb88491_f609_4a24_81f5_d7b0809d79ba.slice/crio-7253f5c0af5f2e1d55d8a43946656d3b63759db7bd98e789020a176a298c8eff WatchSource:0}: Error finding container 7253f5c0af5f2e1d55d8a43946656d3b63759db7bd98e789020a176a298c8eff: Status 404 returned error can't find the container with id 7253f5c0af5f2e1d55d8a43946656d3b63759db7bd98e789020a176a298c8eff
Sep 14 11:06:01.031380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.031345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:01.128369 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.128333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:01.224731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.224636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:01.382150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.382095    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerID="1bc9daa018f941ca778a5cc6dbfe69020c40dbc3015ce33ba25c503880152cb4" exitCode=0
Sep 14 11:06:01.382150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.382142    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerID="1ad59ee42c6bcb6d407d7f145a7de5bcb25189dfa9d702cba330d5c1e7844fe5" exitCode=0
Sep 14 11:06:01.382422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.382308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52" event={"ID":"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a","Type":"ContainerDied","Data":"1bc9daa018f941ca778a5cc6dbfe69020c40dbc3015ce33ba25c503880152cb4"}
Sep 14 11:06:01.382422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.382366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52" event={"ID":"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a","Type":"ContainerDied","Data":"1ad59ee42c6bcb6d407d7f145a7de5bcb25189dfa9d702cba330d5c1e7844fe5"}
Sep 14 11:06:01.384095 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:01.384063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p" event={"ID":"1bb88491-f609-4a24-81f5-d7b0809d79ba","Type":"ContainerStarted","Data":"7253f5c0af5f2e1d55d8a43946656d3b63759db7bd98e789020a176a298c8eff"}
Sep 14 11:06:03.863450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.863422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:06:03.906955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.906931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjmgm\" (UniqueName: \"kubernetes.io/projected/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-kube-api-access-wjmgm\") pod \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") "
Sep 14 11:06:03.907047 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.907030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-util\") pod \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") "
Sep 14 11:06:03.907112 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.907069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-bundle\") pod \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\" (UID: \"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a\") "
Sep 14 11:06:03.907813 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.907781    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-bundle" (OuterVolumeSpecName: "bundle") pod "82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" (UID: "82ca7755-a645-4ff1-ac7b-1f5f33cabe5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:06:03.909226 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.909196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-kube-api-access-wjmgm" (OuterVolumeSpecName: "kube-api-access-wjmgm") pod "82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" (UID: "82ca7755-a645-4ff1-ac7b-1f5f33cabe5a"). InnerVolumeSpecName "kube-api-access-wjmgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:06:03.915657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:03.915517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-util" (OuterVolumeSpecName: "util") pod "82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" (UID: "82ca7755-a645-4ff1-ac7b-1f5f33cabe5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:06:04.008178 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.008146    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:06:04.008178 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.008177    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjmgm\" (UniqueName: \"kubernetes.io/projected/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-kube-api-access-wjmgm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:06:04.008178 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.008188    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82ca7755-a645-4ff1-ac7b-1f5f33cabe5a-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:06:04.407506 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.407473    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52"
Sep 14 11:06:04.407676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.407483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0fdk52" event={"ID":"82ca7755-a645-4ff1-ac7b-1f5f33cabe5a","Type":"ContainerDied","Data":"1f488b89e2f12c5051439dba012ed8c4e7505fc1200881346bfa4ffe47e308b7"}
Sep 14 11:06:04.407676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.407609    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f488b89e2f12c5051439dba012ed8c4e7505fc1200881346bfa4ffe47e308b7"
Sep 14 11:06:04.409192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:04.409162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p" event={"ID":"1bb88491-f609-4a24-81f5-d7b0809d79ba","Type":"ContainerStarted","Data":"a3ba9bb15cb2ca789901afd9038fbadfc0ed5e5e90aaf68c0e0223c3993781ba"}
Sep 14 11:06:05.416290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:05.416258    2563 generic.go:356] "Generic (PLEG): container finished" podID="1bb88491-f609-4a24-81f5-d7b0809d79ba" containerID="a3ba9bb15cb2ca789901afd9038fbadfc0ed5e5e90aaf68c0e0223c3993781ba" exitCode=0
Sep 14 11:06:05.416290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:05.416294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p" event={"ID":"1bb88491-f609-4a24-81f5-d7b0809d79ba","Type":"ContainerDied","Data":"a3ba9bb15cb2ca789901afd9038fbadfc0ed5e5e90aaf68c0e0223c3993781ba"}
Sep 14 11:06:07.427936 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:07.427892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p" event={"ID":"1bb88491-f609-4a24-81f5-d7b0809d79ba","Type":"ContainerStarted","Data":"e95f955f8c83b7359bc8c9db41b391e7a1828cbaf16de02c834d018179345b04"}
Sep 14 11:06:07.428328 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:07.428014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:07.453242 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:07.453140    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p" podStartSLOduration=1.675597599 podStartE2EDuration="7.453121855s" totalImagesPullingTime="4.718268747s" totalInitContainerRuntime="1.059255509s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:07.451266553 +0000 UTC m=+316.073819766" watchObservedRunningTime="2026-09-14 11:06:07.453121855 +0000 UTC m=+316.075675069"
Sep 14 11:06:16.739170 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:16.739139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:17.114455 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:17.114423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:18.435184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.435156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-8xv8p"
Sep 14 11:06:18.766466 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"]
Sep 14 11:06:18.766733 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="util"
Sep 14 11:06:18.766771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766735    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="util"
Sep 14 11:06:18.766771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766745    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="pull"
Sep 14 11:06:18.766771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="pull"
Sep 14 11:06:18.766771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="extract"
Sep 14 11:06:18.766771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="extract"
Sep 14 11:06:18.766920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.766832    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ca7755-a645-4ff1-ac7b-1f5f33cabe5a" containerName="extract"
Sep 14 11:06:18.769971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.769948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.774150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.774123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 11:06:18.774255 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.774123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 11:06:18.774587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.774567    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 11:06:18.774657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.774604    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 11:06:18.775534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.775511    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 11:06:18.775534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.775519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lvnq9\""
Sep 14 11:06:18.792377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.792356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"]
Sep 14 11:06:18.829428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.829379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwkp\" (UniqueName: \"kubernetes.io/projected/de64e813-065d-470a-afa6-9c55de53e4b0-kube-api-access-rdwkp\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.829538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.829440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/de64e813-065d-470a-afa6-9c55de53e4b0-metrics-cert\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.829538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.829516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/de64e813-065d-470a-afa6-9c55de53e4b0-manager-config\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.829637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.829554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de64e813-065d-470a-afa6-9c55de53e4b0-cert\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.930664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.930636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwkp\" (UniqueName: \"kubernetes.io/projected/de64e813-065d-470a-afa6-9c55de53e4b0-kube-api-access-rdwkp\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.930780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.930686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/de64e813-065d-470a-afa6-9c55de53e4b0-metrics-cert\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.930780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.930719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/de64e813-065d-470a-afa6-9c55de53e4b0-manager-config\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.930780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.930746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de64e813-065d-470a-afa6-9c55de53e4b0-cert\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.931355 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.931335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/de64e813-065d-470a-afa6-9c55de53e4b0-manager-config\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.933111 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.933090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de64e813-065d-470a-afa6-9c55de53e4b0-cert\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.933224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.933207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/de64e813-065d-470a-afa6-9c55de53e4b0-metrics-cert\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:18.941523 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:18.941499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwkp\" (UniqueName: \"kubernetes.io/projected/de64e813-065d-470a-afa6-9c55de53e4b0-kube-api-access-rdwkp\") pod \"lws-controller-manager-69cb796bd9-znrrn\" (UID: \"de64e813-065d-470a-afa6-9c55de53e4b0\") " pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:19.080413 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:19.080332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:19.358795 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:19.358769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"]
Sep 14 11:06:19.361827 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:06:19.361801    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde64e813_065d_470a_afa6_9c55de53e4b0.slice/crio-e8056930d7b7c763c6de6b287502c13abcd44c8a0f6e1a617e19deead33c83e9 WatchSource:0}: Error finding container e8056930d7b7c763c6de6b287502c13abcd44c8a0f6e1a617e19deead33c83e9: Status 404 returned error can't find the container with id e8056930d7b7c763c6de6b287502c13abcd44c8a0f6e1a617e19deead33c83e9
Sep 14 11:06:19.495955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:19.495917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn" event={"ID":"de64e813-065d-470a-afa6-9c55de53e4b0","Type":"ContainerStarted","Data":"e8056930d7b7c763c6de6b287502c13abcd44c8a0f6e1a617e19deead33c83e9"}
Sep 14 11:06:20.519543 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:20.519505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:21.199734 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.199700    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"]
Sep 14 11:06:21.203700 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.203677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.211450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.211423    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 11:06:21.211586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.211469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 14 11:06:21.211816 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.211773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 14 11:06:21.211922 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.211814    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 11:06:21.214751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.214732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vft58\""
Sep 14 11:06:21.231290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.231252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"]
Sep 14 11:06:21.252612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.252570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8005ec9-a549-4e2b-8582-715150557301-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.252796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.252633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wg8r\" (UniqueName: \"kubernetes.io/projected/c8005ec9-a549-4e2b-8582-715150557301-kube-api-access-6wg8r\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.252796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.252680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8005ec9-a549-4e2b-8582-715150557301-tmp\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.354028 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.353989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8005ec9-a549-4e2b-8582-715150557301-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.354414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.354039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wg8r\" (UniqueName: \"kubernetes.io/projected/c8005ec9-a549-4e2b-8582-715150557301-kube-api-access-6wg8r\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.354414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.354083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8005ec9-a549-4e2b-8582-715150557301-tmp\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.354414 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:06:21.354158    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 14 11:06:21.354414 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:06:21.354264    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8005ec9-a549-4e2b-8582-715150557301-tls-certs podName:c8005ec9-a549-4e2b-8582-715150557301 nodeName:}" failed. No retries permitted until 2026-09-14 11:06:21.854229721 +0000 UTC m=+330.476782964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c8005ec9-a549-4e2b-8582-715150557301-tls-certs") pod "kube-auth-proxy-5f95fc98f9-7bj84" (UID: "c8005ec9-a549-4e2b-8582-715150557301") : secret "kube-auth-proxy-tls" not found
Sep 14 11:06:21.356951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.356926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8005ec9-a549-4e2b-8582-715150557301-tmp\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.371952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.371908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wg8r\" (UniqueName: \"kubernetes.io/projected/c8005ec9-a549-4e2b-8582-715150557301-kube-api-access-6wg8r\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.857338 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.857301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8005ec9-a549-4e2b-8582-715150557301-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:21.859732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:21.859705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8005ec9-a549-4e2b-8582-715150557301-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-7bj84\" (UID: \"c8005ec9-a549-4e2b-8582-715150557301\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:22.113245 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.113159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:22.117110 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.117093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"
Sep 14 11:06:22.396548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.396518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84"]
Sep 14 11:06:22.398721 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:06:22.398673    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8005ec9_a549_4e2b_8582_715150557301.slice/crio-509a2d06b83abbbe5044f1a7c63f57fdfc846d01b3f95cad10474e24a8b11ad8 WatchSource:0}: Error finding container 509a2d06b83abbbe5044f1a7c63f57fdfc846d01b3f95cad10474e24a8b11ad8: Status 404 returned error can't find the container with id 509a2d06b83abbbe5044f1a7c63f57fdfc846d01b3f95cad10474e24a8b11ad8
Sep 14 11:06:22.512168 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.512133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84" event={"ID":"c8005ec9-a549-4e2b-8582-715150557301","Type":"ContainerStarted","Data":"509a2d06b83abbbe5044f1a7c63f57fdfc846d01b3f95cad10474e24a8b11ad8"}
Sep 14 11:06:22.513775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.513750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn" event={"ID":"de64e813-065d-470a-afa6-9c55de53e4b0","Type":"ContainerStarted","Data":"6df2b62921f1ba985f1bcb3cc6fa9641462f8ac5ba92f30dac0470ab0bc4c8db"}
Sep 14 11:06:22.513927 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.513915    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:22.546960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.546907    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn" podStartSLOduration=2.388924076 podStartE2EDuration="4.54688981s" totalImagesPullingTime="2.157965734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:22.54614783 +0000 UTC m=+331.168701040" watchObservedRunningTime="2026-09-14 11:06:22.54688981 +0000 UTC m=+331.169443024"
Sep 14 11:06:22.956530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:22.956497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:23.068896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:23.068728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:24.566468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:24.566427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:25.533189 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:25.533100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84" event={"ID":"c8005ec9-a549-4e2b-8582-715150557301","Type":"ContainerStarted","Data":"8db11e6ccee68e64525812b4f42fe05b8554659899a65b355c2dbe0ccd4195d6"}
Sep 14 11:06:25.569006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:25.568950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-7bj84" podStartSLOduration=1.742652026 podStartE2EDuration="4.568932853s" totalImagesPullingTime="2.826280827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:25.567014853 +0000 UTC m=+334.189568066" watchObservedRunningTime="2026-09-14 11:06:25.568932853 +0000 UTC m=+334.191486067"
Sep 14 11:06:25.691784 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:25.691746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:25.789612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:25.789529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:28.620551 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:28.620497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:28.714363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:28.714330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:30.353866 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:30.353827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:30.591750 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:30.591651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:31.817274 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:31.817227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:31.920412 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:31.920362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:32.011336 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:32.011289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:32.215169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:32.215138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:32.294374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:32.294341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:32.442266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:32.442232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:33.523768 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:33.523741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69cb796bd9-znrrn"
Sep 14 11:06:43.954891 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:43.954858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:44.243066 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:44.242922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:44.750168 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:44.750133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:44.945971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:44.945932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:45.236019 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:45.235990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:45.438962 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:45.438932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:45.605943 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:45.605911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:57.038218 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:57.038187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:59.076145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:59.076111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:06:59.137314 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:06:59.137285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:32.590314 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:32.590271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:32.676879 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:32.676846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:37.473704 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.473662    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"]
Sep 14 11:07:37.477340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.477321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.481721 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.481693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sz8ts\""
Sep 14 11:07:37.481839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.481762    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 11:07:37.481839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.481787    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 11:07:37.493793 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.493765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"]
Sep 14 11:07:37.589064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.589026    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"]
Sep 14 11:07:37.592731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.592712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.602643 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.602614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.602772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.602686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.602772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.602753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9qs\" (UniqueName: \"kubernetes.io/projected/5ca1c958-bc33-4375-8883-abc8af9b9ad1-kube-api-access-md9qs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.603416 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.603371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"]
Sep 14 11:07:37.684735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.684697    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"]
Sep 14 11:07:37.688323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.688298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.700573 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.700545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"]
Sep 14 11:07:37.703485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmtr\" (UniqueName: \"kubernetes.io/projected/b688985b-7e63-492c-a030-d8f661342d87-kube-api-access-rtmtr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.703627 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.703627 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.703627 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.703799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9qs\" (UniqueName: \"kubernetes.io/projected/5ca1c958-bc33-4375-8883-abc8af9b9ad1-kube-api-access-md9qs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.703799 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.703909 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.703870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.704041 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.704023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.713569 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.713545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9qs\" (UniqueName: \"kubernetes.io/projected/5ca1c958-bc33-4375-8883-abc8af9b9ad1-kube-api-access-md9qs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.787094 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.786989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:37.804377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvc5\" (UniqueName: \"kubernetes.io/projected/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-kube-api-access-tpvc5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.804513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmtr\" (UniqueName: \"kubernetes.io/projected/b688985b-7e63-492c-a030-d8f661342d87-kube-api-access-rtmtr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.804513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.804513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804496    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"]
Sep 14 11:07:37.804674 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.804728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.804786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.804851 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.804830    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.805037 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.805018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.809309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.809288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:37.813578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.813555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmtr\" (UniqueName: \"kubernetes.io/projected/b688985b-7e63-492c-a030-d8f661342d87-kube-api-access-rtmtr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.823873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.823841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"]
Sep 14 11:07:37.902649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.902620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:37.905213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:37.905358 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:37.905358 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.905508 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8b5d\" (UniqueName: \"kubernetes.io/projected/68af827b-f7ad-4817-b9d7-60b1a87360f6-kube-api-access-j8b5d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:37.905508 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.905695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvc5\" (UniqueName: \"kubernetes.io/projected/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-kube-api-access-tpvc5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.905695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.905887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.905866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.915635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.915614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvc5\" (UniqueName: \"kubernetes.io/projected/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-kube-api-access-tpvc5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:37.998708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:37.998672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:38.007062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.007037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.007151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.007081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.007191 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.007154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8b5d\" (UniqueName: \"kubernetes.io/projected/68af827b-f7ad-4817-b9d7-60b1a87360f6-kube-api-access-j8b5d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.007483 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.007461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.007483 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.007472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.016539 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.016519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8b5d\" (UniqueName: \"kubernetes.io/projected/68af827b-f7ad-4817-b9d7-60b1a87360f6-kube-api-access-j8b5d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.073823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.073670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"]
Sep 14 11:07:38.076458 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:07:38.076429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca1c958_bc33_4375_8883_abc8af9b9ad1.slice/crio-6c7b979b9c43466e35f26841acade641a0e870dbca9fba302d4aaa7239799c82 WatchSource:0}: Error finding container 6c7b979b9c43466e35f26841acade641a0e870dbca9fba302d4aaa7239799c82: Status 404 returned error can't find the container with id 6c7b979b9c43466e35f26841acade641a0e870dbca9fba302d4aaa7239799c82
Sep 14 11:07:38.129104 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.129081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:38.182721 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.182567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"]
Sep 14 11:07:38.226278 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:07:38.226245    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb688985b_7e63_492c_a030_d8f661342d87.slice/crio-c49565fe39ec3d8c202675abc1716b8cbbf3c9395f7db69a51f1dff90eed7961 WatchSource:0}: Error finding container c49565fe39ec3d8c202675abc1716b8cbbf3c9395f7db69a51f1dff90eed7961: Status 404 returned error can't find the container with id c49565fe39ec3d8c202675abc1716b8cbbf3c9395f7db69a51f1dff90eed7961
Sep 14 11:07:38.280836 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.280815    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"]
Sep 14 11:07:38.282410 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:07:38.282349    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f86c7_9d51_4ea6_80cf_6aa9b9df4e30.slice/crio-82657502370ac3fe8dce2fb5c35707eeea71172f91882f28d63778a0d02f4ef6 WatchSource:0}: Error finding container 82657502370ac3fe8dce2fb5c35707eeea71172f91882f28d63778a0d02f4ef6: Status 404 returned error can't find the container with id 82657502370ac3fe8dce2fb5c35707eeea71172f91882f28d63778a0d02f4ef6
Sep 14 11:07:38.402352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.402305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"]
Sep 14 11:07:38.445203 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:07:38.445175    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68af827b_f7ad_4817_b9d7_60b1a87360f6.slice/crio-70bafaa9575439546073752a0876fa3b9c5e24d34c93fb023e094fa53ea11951 WatchSource:0}: Error finding container 70bafaa9575439546073752a0876fa3b9c5e24d34c93fb023e094fa53ea11951: Status 404 returned error can't find the container with id 70bafaa9575439546073752a0876fa3b9c5e24d34c93fb023e094fa53ea11951
Sep 14 11:07:38.900892 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.900859    2563 generic.go:356] "Generic (PLEG): container finished" podID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerID="437d71c8f2197dbc92cfc1213e7e7df296b09ae912fba759347a55452acb23b2" exitCode=0
Sep 14 11:07:38.901346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.900948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl" event={"ID":"68af827b-f7ad-4817-b9d7-60b1a87360f6","Type":"ContainerDied","Data":"437d71c8f2197dbc92cfc1213e7e7df296b09ae912fba759347a55452acb23b2"}
Sep 14 11:07:38.901346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.901017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl" event={"ID":"68af827b-f7ad-4817-b9d7-60b1a87360f6","Type":"ContainerStarted","Data":"70bafaa9575439546073752a0876fa3b9c5e24d34c93fb023e094fa53ea11951"}
Sep 14 11:07:38.902668 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.902645    2563 generic.go:356] "Generic (PLEG): container finished" podID="b688985b-7e63-492c-a030-d8f661342d87" containerID="38446d9629fb6302bd7cb814bc31a5c1b3487ac05a710de1c4b4227ec3e4730f" exitCode=0
Sep 14 11:07:38.902769 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.902731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7" event={"ID":"b688985b-7e63-492c-a030-d8f661342d87","Type":"ContainerDied","Data":"38446d9629fb6302bd7cb814bc31a5c1b3487ac05a710de1c4b4227ec3e4730f"}
Sep 14 11:07:38.902829 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.902779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7" event={"ID":"b688985b-7e63-492c-a030-d8f661342d87","Type":"ContainerStarted","Data":"c49565fe39ec3d8c202675abc1716b8cbbf3c9395f7db69a51f1dff90eed7961"}
Sep 14 11:07:38.904482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.904459    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerID="c88080e2bdd05ab09bc68bb0dbe55bf7127f986a7289f5f22a711880eb9767b8" exitCode=0
Sep 14 11:07:38.904728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.904540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l" event={"ID":"5ca1c958-bc33-4375-8883-abc8af9b9ad1","Type":"ContainerDied","Data":"c88080e2bdd05ab09bc68bb0dbe55bf7127f986a7289f5f22a711880eb9767b8"}
Sep 14 11:07:38.904728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.904581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l" event={"ID":"5ca1c958-bc33-4375-8883-abc8af9b9ad1","Type":"ContainerStarted","Data":"6c7b979b9c43466e35f26841acade641a0e870dbca9fba302d4aaa7239799c82"}
Sep 14 11:07:38.906501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.906479    2563 generic.go:356] "Generic (PLEG): container finished" podID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerID="bccdf8e55a1aaef560eac710f766334b03e5f8a145d3e186a51b0e431f1ecee6" exitCode=0
Sep 14 11:07:38.906587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.906544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858" event={"ID":"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30","Type":"ContainerDied","Data":"bccdf8e55a1aaef560eac710f766334b03e5f8a145d3e186a51b0e431f1ecee6"}
Sep 14 11:07:38.906587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:38.906582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858" event={"ID":"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30","Type":"ContainerStarted","Data":"82657502370ac3fe8dce2fb5c35707eeea71172f91882f28d63778a0d02f4ef6"}
Sep 14 11:07:39.914361 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:39.914330    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerID="ee466044042e427b5a40435ca126d77dfde7201bbff7b9dc6c3ec60b3db21b98" exitCode=0
Sep 14 11:07:39.916622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:39.916595    2563 generic.go:356] "Generic (PLEG): container finished" podID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerID="9a85f981b8708256711791a0f116634dc893b04c32d49d813e9f86803b8b8778" exitCode=0
Sep 14 11:07:39.918963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:39.918930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l" event={"ID":"5ca1c958-bc33-4375-8883-abc8af9b9ad1","Type":"ContainerDied","Data":"ee466044042e427b5a40435ca126d77dfde7201bbff7b9dc6c3ec60b3db21b98"}
Sep 14 11:07:39.919057 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:39.918982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl" event={"ID":"68af827b-f7ad-4817-b9d7-60b1a87360f6","Type":"ContainerDied","Data":"9a85f981b8708256711791a0f116634dc893b04c32d49d813e9f86803b8b8778"}
Sep 14 11:07:40.923656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:40.923622    2563 generic.go:356] "Generic (PLEG): container finished" podID="b688985b-7e63-492c-a030-d8f661342d87" containerID="f62668ed0b899ec505a01c1af1a1a1a4bee41b0093f0150aecb523d0d0a3ecd2" exitCode=0
Sep 14 11:07:40.924049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:40.923711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7" event={"ID":"b688985b-7e63-492c-a030-d8f661342d87","Type":"ContainerDied","Data":"f62668ed0b899ec505a01c1af1a1a1a4bee41b0093f0150aecb523d0d0a3ecd2"}
Sep 14 11:07:40.926110 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:40.926087    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerID="aedd1e46446e485c2913daf2f83cabe8e6238be8229615a439a2753195b046ae" exitCode=0
Sep 14 11:07:40.926212 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:40.926185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l" event={"ID":"5ca1c958-bc33-4375-8883-abc8af9b9ad1","Type":"ContainerDied","Data":"aedd1e46446e485c2913daf2f83cabe8e6238be8229615a439a2753195b046ae"}
Sep 14 11:07:40.928286 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:40.928265    2563 generic.go:356] "Generic (PLEG): container finished" podID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerID="f48115e27bb2fe7c39a9da65a40a7a146d8e439b33d2658093febe09fd745080" exitCode=0
Sep 14 11:07:40.928375 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:40.928294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl" event={"ID":"68af827b-f7ad-4817-b9d7-60b1a87360f6","Type":"ContainerDied","Data":"f48115e27bb2fe7c39a9da65a40a7a146d8e439b33d2658093febe09fd745080"}
Sep 14 11:07:41.935101 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:41.935067    2563 generic.go:356] "Generic (PLEG): container finished" podID="b688985b-7e63-492c-a030-d8f661342d87" containerID="94c6bccd1e22e0ceac2c522b6a2db1c6a6dab348ec1857eeafc21a0b7cc95220" exitCode=0
Sep 14 11:07:41.935536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:41.935152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7" event={"ID":"b688985b-7e63-492c-a030-d8f661342d87","Type":"ContainerDied","Data":"94c6bccd1e22e0ceac2c522b6a2db1c6a6dab348ec1857eeafc21a0b7cc95220"}
Sep 14 11:07:42.099812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.099774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:42.101740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.101721    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:42.245224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245126    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-util\") pod \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") "
Sep 14 11:07:42.245224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8b5d\" (UniqueName: \"kubernetes.io/projected/68af827b-f7ad-4817-b9d7-60b1a87360f6-kube-api-access-j8b5d\") pod \"68af827b-f7ad-4817-b9d7-60b1a87360f6\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") "
Sep 14 11:07:42.245469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-bundle\") pod \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") "
Sep 14 11:07:42.245469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-bundle\") pod \"68af827b-f7ad-4817-b9d7-60b1a87360f6\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") "
Sep 14 11:07:42.245469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-util\") pod \"68af827b-f7ad-4817-b9d7-60b1a87360f6\" (UID: \"68af827b-f7ad-4817-b9d7-60b1a87360f6\") "
Sep 14 11:07:42.245469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md9qs\" (UniqueName: \"kubernetes.io/projected/5ca1c958-bc33-4375-8883-abc8af9b9ad1-kube-api-access-md9qs\") pod \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\" (UID: \"5ca1c958-bc33-4375-8883-abc8af9b9ad1\") "
Sep 14 11:07:42.245837 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.245812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-bundle" (OuterVolumeSpecName: "bundle") pod "5ca1c958-bc33-4375-8883-abc8af9b9ad1" (UID: "5ca1c958-bc33-4375-8883-abc8af9b9ad1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:42.246081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.246047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-bundle" (OuterVolumeSpecName: "bundle") pod "68af827b-f7ad-4817-b9d7-60b1a87360f6" (UID: "68af827b-f7ad-4817-b9d7-60b1a87360f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:42.247755 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.247728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68af827b-f7ad-4817-b9d7-60b1a87360f6-kube-api-access-j8b5d" (OuterVolumeSpecName: "kube-api-access-j8b5d") pod "68af827b-f7ad-4817-b9d7-60b1a87360f6" (UID: "68af827b-f7ad-4817-b9d7-60b1a87360f6"). InnerVolumeSpecName "kube-api-access-j8b5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:07:42.247847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.247827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca1c958-bc33-4375-8883-abc8af9b9ad1-kube-api-access-md9qs" (OuterVolumeSpecName: "kube-api-access-md9qs") pod "5ca1c958-bc33-4375-8883-abc8af9b9ad1" (UID: "5ca1c958-bc33-4375-8883-abc8af9b9ad1"). InnerVolumeSpecName "kube-api-access-md9qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:07:42.251793 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.251770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-util" (OuterVolumeSpecName: "util") pod "5ca1c958-bc33-4375-8883-abc8af9b9ad1" (UID: "5ca1c958-bc33-4375-8883-abc8af9b9ad1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:42.253492 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.253469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-util" (OuterVolumeSpecName: "util") pod "68af827b-f7ad-4817-b9d7-60b1a87360f6" (UID: "68af827b-f7ad-4817-b9d7-60b1a87360f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:42.346309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.346273    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:42.346309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.346300    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8b5d\" (UniqueName: \"kubernetes.io/projected/68af827b-f7ad-4817-b9d7-60b1a87360f6-kube-api-access-j8b5d\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:42.346309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.346311    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ca1c958-bc33-4375-8883-abc8af9b9ad1-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:42.346309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.346320    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:42.346601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.346327    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68af827b-f7ad-4817-b9d7-60b1a87360f6-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:42.346601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.346335    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md9qs\" (UniqueName: \"kubernetes.io/projected/5ca1c958-bc33-4375-8883-abc8af9b9ad1-kube-api-access-md9qs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:42.942554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.942519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l"
Sep 14 11:07:42.942554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.942536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64tp5l" event={"ID":"5ca1c958-bc33-4375-8883-abc8af9b9ad1","Type":"ContainerDied","Data":"6c7b979b9c43466e35f26841acade641a0e870dbca9fba302d4aaa7239799c82"}
Sep 14 11:07:42.942992 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.942586    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7b979b9c43466e35f26841acade641a0e870dbca9fba302d4aaa7239799c82"
Sep 14 11:07:42.944547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.944529    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl"
Sep 14 11:07:42.944686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.944573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847affvnl" event={"ID":"68af827b-f7ad-4817-b9d7-60b1a87360f6","Type":"ContainerDied","Data":"70bafaa9575439546073752a0876fa3b9c5e24d34c93fb023e094fa53ea11951"}
Sep 14 11:07:42.944686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:42.944612    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70bafaa9575439546073752a0876fa3b9c5e24d34c93fb023e094fa53ea11951"
Sep 14 11:07:43.072506 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.072478    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:43.152501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.152467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-bundle\") pod \"b688985b-7e63-492c-a030-d8f661342d87\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") "
Sep 14 11:07:43.152663 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.152564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtmtr\" (UniqueName: \"kubernetes.io/projected/b688985b-7e63-492c-a030-d8f661342d87-kube-api-access-rtmtr\") pod \"b688985b-7e63-492c-a030-d8f661342d87\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") "
Sep 14 11:07:43.152663 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.152600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-util\") pod \"b688985b-7e63-492c-a030-d8f661342d87\" (UID: \"b688985b-7e63-492c-a030-d8f661342d87\") "
Sep 14 11:07:43.153088 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.153051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-bundle" (OuterVolumeSpecName: "bundle") pod "b688985b-7e63-492c-a030-d8f661342d87" (UID: "b688985b-7e63-492c-a030-d8f661342d87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:43.154788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.154758    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b688985b-7e63-492c-a030-d8f661342d87-kube-api-access-rtmtr" (OuterVolumeSpecName: "kube-api-access-rtmtr") pod "b688985b-7e63-492c-a030-d8f661342d87" (UID: "b688985b-7e63-492c-a030-d8f661342d87"). InnerVolumeSpecName "kube-api-access-rtmtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:07:43.157887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.157860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-util" (OuterVolumeSpecName: "util") pod "b688985b-7e63-492c-a030-d8f661342d87" (UID: "b688985b-7e63-492c-a030-d8f661342d87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:43.253809 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.253738    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmtr\" (UniqueName: \"kubernetes.io/projected/b688985b-7e63-492c-a030-d8f661342d87-kube-api-access-rtmtr\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:43.253809 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.253763    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:43.253809 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.253773    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b688985b-7e63-492c-a030-d8f661342d87-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:43.952019 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.951990    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7"
Sep 14 11:07:43.952019 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.952007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvzhw7" event={"ID":"b688985b-7e63-492c-a030-d8f661342d87","Type":"ContainerDied","Data":"c49565fe39ec3d8c202675abc1716b8cbbf3c9395f7db69a51f1dff90eed7961"}
Sep 14 11:07:43.952450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:43.952052    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c49565fe39ec3d8c202675abc1716b8cbbf3c9395f7db69a51f1dff90eed7961"
Sep 14 11:07:45.231534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:45.231473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:45.290236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:45.290199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:50.988647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:50.988613    2563 generic.go:356] "Generic (PLEG): container finished" podID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerID="951b90f8ec7b14af0ea67db675702ccab3e6e50167a3c4f74539fd28023c7a9e" exitCode=0
Sep 14 11:07:50.988647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:50.988653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858" event={"ID":"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30","Type":"ContainerDied","Data":"951b90f8ec7b14af0ea67db675702ccab3e6e50167a3c4f74539fd28023c7a9e"}
Sep 14 11:07:51.995978 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:51.995939    2563 generic.go:356] "Generic (PLEG): container finished" podID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerID="a235cbe23e93d10bebdb8b21027797452be3948649860fcff7876b5275797c2e" exitCode=0
Sep 14 11:07:51.996438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:51.996029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858" event={"ID":"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30","Type":"ContainerDied","Data":"a235cbe23e93d10bebdb8b21027797452be3948649860fcff7876b5275797c2e"}
Sep 14 11:07:53.127065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.127039    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:53.237494 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.237458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpvc5\" (UniqueName: \"kubernetes.io/projected/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-kube-api-access-tpvc5\") pod \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") "
Sep 14 11:07:53.237688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.237583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-bundle\") pod \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") "
Sep 14 11:07:53.237688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.237616    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-util\") pod \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\" (UID: \"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30\") "
Sep 14 11:07:53.238174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.238147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-bundle" (OuterVolumeSpecName: "bundle") pod "708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" (UID: "708f86c7-9d51-4ea6-80cf-6aa9b9df4e30"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:53.239450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.239423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-kube-api-access-tpvc5" (OuterVolumeSpecName: "kube-api-access-tpvc5") pod "708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" (UID: "708f86c7-9d51-4ea6-80cf-6aa9b9df4e30"). InnerVolumeSpecName "kube-api-access-tpvc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:07:53.242301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.242274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-util" (OuterVolumeSpecName: "util") pod "708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" (UID: "708f86c7-9d51-4ea6-80cf-6aa9b9df4e30"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:07:53.338656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.338571    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:53.338656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.338601    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-util\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:53.338656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:53.338610    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvc5\" (UniqueName: \"kubernetes.io/projected/708f86c7-9d51-4ea6-80cf-6aa9b9df4e30-kube-api-access-tpvc5\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:07:54.008317 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:54.008286    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858"
Sep 14 11:07:54.008499 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:54.008315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wk858" event={"ID":"708f86c7-9d51-4ea6-80cf-6aa9b9df4e30","Type":"ContainerDied","Data":"82657502370ac3fe8dce2fb5c35707eeea71172f91882f28d63778a0d02f4ef6"}
Sep 14 11:07:54.008499 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:54.008365    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82657502370ac3fe8dce2fb5c35707eeea71172f91882f28d63778a0d02f4ef6"
Sep 14 11:07:55.808289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:55.808255    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbdccf59c-qx9zw"]
Sep 14 11:07:55.921806 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:55.921772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:57.417148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:57.417112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:07:57.480578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:07:57.480544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:03.380080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:03.380043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:04.187097 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:04.187049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:07.264607 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:07.264570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:07.368738 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:07.368706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:07.468382 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:07.468343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:09.425047 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:09.425007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:09.603244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:09.603205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:11.918475 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:11.918419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:12.393488 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:12.393454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:12.712567 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:12.712481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:12.862547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:12.862517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:12.978363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:12.978280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:13.777227 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:13.777190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:14.890183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:14.890108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:14.981850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:14.981820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:19.791045 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:19.791011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:19.890986 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:19.890954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:20.208841 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:20.208788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:20.303407 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:20.303364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:20.828179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:20.828141    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cbdccf59c-qx9zw" podUID="17da9208-bb8d-4d11-a042-8f4955cbe121" containerName="console" containerID="cri-o://8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41" gracePeriod=15
Sep 14 11:08:21.080944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.080882    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:08:21.156015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.155979    2563 generic.go:356] "Generic (PLEG): container finished" podID="17da9208-bb8d-4d11-a042-8f4955cbe121" containerID="8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41" exitCode=0
Sep 14 11:08:21.156188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.156032    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbdccf59c-qx9zw"
Sep 14 11:08:21.156188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.156061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdccf59c-qx9zw" event={"ID":"17da9208-bb8d-4d11-a042-8f4955cbe121","Type":"ContainerDied","Data":"8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41"}
Sep 14 11:08:21.156188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.156109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbdccf59c-qx9zw" event={"ID":"17da9208-bb8d-4d11-a042-8f4955cbe121","Type":"ContainerDied","Data":"85cbb900d9899fad005410e02bf151133d71d0a559dbf2ebe9eb87c3f581bf76"}
Sep 14 11:08:21.156188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.156133    2563 scope.go:117] "RemoveContainer" containerID="8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41"
Sep 14 11:08:21.165003 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.164976    2563 scope.go:117] "RemoveContainer" containerID="8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41"
Sep 14 11:08:21.165291 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:08:21.165274    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41\": container with ID starting with 8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41 not found: ID does not exist" containerID="8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41"
Sep 14 11:08:21.165344 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.165299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41"} err="failed to get container status \"8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41\": rpc error: code = NotFound desc = could not find container \"8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41\": container with ID starting with 8c76e302e93c6372579fee7bd075d0aedda79a26f564a55ded3048cad1337e41 not found: ID does not exist"
Sep 14 11:08:21.170188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-oauth-config\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170308 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-oauth-serving-cert\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170308 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-serving-cert\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170308 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjqd4\" (UniqueName: \"kubernetes.io/projected/17da9208-bb8d-4d11-a042-8f4955cbe121-kube-api-access-vjqd4\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170308 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-console-config\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170549 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-service-ca\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170549 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-trusted-ca-bundle\") pod \"17da9208-bb8d-4d11-a042-8f4955cbe121\" (UID: \"17da9208-bb8d-4d11-a042-8f4955cbe121\") "
Sep 14 11:08:21.170731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:08:21.170841 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170734    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-service-ca" (OuterVolumeSpecName: "service-ca") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:08:21.170905 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:08:21.170967 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.170933    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-console-config" (OuterVolumeSpecName: "console-config") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:08:21.172414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.172365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:08:21.172483 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.172431    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:08:21.172540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.172523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17da9208-bb8d-4d11-a042-8f4955cbe121-kube-api-access-vjqd4" (OuterVolumeSpecName: "kube-api-access-vjqd4") pod "17da9208-bb8d-4d11-a042-8f4955cbe121" (UID: "17da9208-bb8d-4d11-a042-8f4955cbe121"). InnerVolumeSpecName "kube-api-access-vjqd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:08:21.271410 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271363    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-oauth-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.271410 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271414    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-serving-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.271604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271426    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjqd4\" (UniqueName: \"kubernetes.io/projected/17da9208-bb8d-4d11-a042-8f4955cbe121-kube-api-access-vjqd4\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.271604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271435    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-console-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.271604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271444    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.271604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271451    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17da9208-bb8d-4d11-a042-8f4955cbe121-trusted-ca-bundle\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.271604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.271459    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17da9208-bb8d-4d11-a042-8f4955cbe121-console-oauth-config\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:08:21.480724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.480693    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbdccf59c-qx9zw"]
Sep 14 11:08:21.500097 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.500065    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbdccf59c-qx9zw"]
Sep 14 11:08:21.918324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:21.918292    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17da9208-bb8d-4d11-a042-8f4955cbe121" path="/var/lib/kubelet/pods/17da9208-bb8d-4d11-a042-8f4955cbe121/volumes"
Sep 14 11:08:23.179794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:23.179762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:23.282154 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:23.282117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:25.397719 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.397670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:25.497536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.497504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:25.576046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"]
Sep 14 11:08:25.576360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576348    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="util"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576362    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="util"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576374    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="pull"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576379    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="pull"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576404    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="util"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576410    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="util"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576418    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="extract"
Sep 14 11:08:25.576421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576431    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576436    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576447    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17da9208-bb8d-4d11-a042-8f4955cbe121" containerName="console"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576452    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17da9208-bb8d-4d11-a042-8f4955cbe121" containerName="console"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576459    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="util"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="util"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="pull"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576473    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="pull"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576482    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576487    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="pull"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576497    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="pull"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576502    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="pull"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576508    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="pull"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576514    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576518    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576524    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="util"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576529    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="util"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576578    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17da9208-bb8d-4d11-a042-8f4955cbe121" containerName="console"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576586    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68af827b-f7ad-4817-b9d7-60b1a87360f6" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576595    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca1c958-bc33-4375-8883-abc8af9b9ad1" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576602    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="708f86c7-9d51-4ea6-80cf-6aa9b9df4e30" containerName="extract"
Sep 14 11:08:25.576637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.576609    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b688985b-7e63-492c-a030-d8f661342d87" containerName="extract"
Sep 14 11:08:25.581084 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.581065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:25.584138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.584117    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 11:08:25.584548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.584525    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 11:08:25.584701 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.584688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v5pz5\""
Sep 14 11:08:25.584752 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.584713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 14 11:08:25.598709 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.598686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"]
Sep 14 11:08:25.711775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.711690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgzs\" (UniqueName: \"kubernetes.io/projected/99166d86-7ea2-476e-adfb-cd36e574dd94-kube-api-access-qpgzs\") pod \"dns-operator-controller-manager-7b84cd7c9c-sxr4b\" (UID: \"99166d86-7ea2-476e-adfb-cd36e574dd94\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:25.812960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.812923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgzs\" (UniqueName: \"kubernetes.io/projected/99166d86-7ea2-476e-adfb-cd36e574dd94-kube-api-access-qpgzs\") pod \"dns-operator-controller-manager-7b84cd7c9c-sxr4b\" (UID: \"99166d86-7ea2-476e-adfb-cd36e574dd94\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:25.833918 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.833884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgzs\" (UniqueName: \"kubernetes.io/projected/99166d86-7ea2-476e-adfb-cd36e574dd94-kube-api-access-qpgzs\") pod \"dns-operator-controller-manager-7b84cd7c9c-sxr4b\" (UID: \"99166d86-7ea2-476e-adfb-cd36e574dd94\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:25.890808 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:25.890769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:26.178867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:26.178829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"]
Sep 14 11:08:26.184321 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:08:26.184256    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99166d86_7ea2_476e_adfb_cd36e574dd94.slice/crio-393d47ad7fe336b4b831eaa0ec6359aa8a97bf97dc8b1a296a7d5ff16974977a WatchSource:0}: Error finding container 393d47ad7fe336b4b831eaa0ec6359aa8a97bf97dc8b1a296a7d5ff16974977a: Status 404 returned error can't find the container with id 393d47ad7fe336b4b831eaa0ec6359aa8a97bf97dc8b1a296a7d5ff16974977a
Sep 14 11:08:27.193753 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:27.193714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b" event={"ID":"99166d86-7ea2-476e-adfb-cd36e574dd94","Type":"ContainerStarted","Data":"393d47ad7fe336b4b831eaa0ec6359aa8a97bf97dc8b1a296a7d5ff16974977a"}
Sep 14 11:08:30.906789 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.906754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:30.915442 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.915413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"]
Sep 14 11:08:30.920179 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.920160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:30.925509 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.925484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 14 11:08:30.925630 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.925491    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 14 11:08:30.925630 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.925524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7khrq\""
Sep 14 11:08:30.931963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:30.931941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"]
Sep 14 11:08:31.061318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.061278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kh2b\" (UniqueName: \"kubernetes.io/projected/13b3ece8-bb0a-4950-9af4-74e28a9bb928-kube-api-access-8kh2b\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.061505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.061351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13b3ece8-bb0a-4950-9af4-74e28a9bb928-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.061505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.061430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b3ece8-bb0a-4950-9af4-74e28a9bb928-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.162150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.162063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kh2b\" (UniqueName: \"kubernetes.io/projected/13b3ece8-bb0a-4950-9af4-74e28a9bb928-kube-api-access-8kh2b\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.162150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.162128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13b3ece8-bb0a-4950-9af4-74e28a9bb928-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.162340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.162158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b3ece8-bb0a-4950-9af4-74e28a9bb928-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.162378 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:08:31.162336    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 14 11:08:31.162440 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:08:31.162421    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b3ece8-bb0a-4950-9af4-74e28a9bb928-plugin-serving-cert podName:13b3ece8-bb0a-4950-9af4-74e28a9bb928 nodeName:}" failed. No retries permitted until 2026-09-14 11:08:31.662376132 +0000 UTC m=+460.284929326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/13b3ece8-bb0a-4950-9af4-74e28a9bb928-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-mpr2j" (UID: "13b3ece8-bb0a-4950-9af4-74e28a9bb928") : secret "plugin-serving-cert" not found
Sep 14 11:08:31.163031 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.163009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13b3ece8-bb0a-4950-9af4-74e28a9bb928-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.170902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.170881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kh2b\" (UniqueName: \"kubernetes.io/projected/13b3ece8-bb0a-4950-9af4-74e28a9bb928-kube-api-access-8kh2b\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.288884 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.288853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:31.384319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.384255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:31.497584 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.497485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:31.667098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.667068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b3ece8-bb0a-4950-9af4-74e28a9bb928-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.669563 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.669543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b3ece8-bb0a-4950-9af4-74e28a9bb928-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mpr2j\" (UID: \"13b3ece8-bb0a-4950-9af4-74e28a9bb928\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:31.723481 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.723447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:31.830098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:31.830011    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"
Sep 14 11:08:32.106611 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:32.106583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j"]
Sep 14 11:08:32.109575 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:08:32.109549    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13b3ece8_bb0a_4950_9af4_74e28a9bb928.slice/crio-a78b24c4b9a554faa877a3f28f35a121b3be2dc27de70427279672495dfc3503 WatchSource:0}: Error finding container a78b24c4b9a554faa877a3f28f35a121b3be2dc27de70427279672495dfc3503: Status 404 returned error can't find the container with id a78b24c4b9a554faa877a3f28f35a121b3be2dc27de70427279672495dfc3503
Sep 14 11:08:32.222619 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:32.222587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j" event={"ID":"13b3ece8-bb0a-4950-9af4-74e28a9bb928","Type":"ContainerStarted","Data":"a78b24c4b9a554faa877a3f28f35a121b3be2dc27de70427279672495dfc3503"}
Sep 14 11:08:39.267464 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.267430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b" event={"ID":"99166d86-7ea2-476e-adfb-cd36e574dd94","Type":"ContainerStarted","Data":"fa0a24186c84dc4cba8d2f8867f0426b5b74cf069fcd717d5dcf7f1d55bb8bfb"}
Sep 14 11:08:39.267877 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.267543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:39.269126 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.269105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j" event={"ID":"13b3ece8-bb0a-4950-9af4-74e28a9bb928","Type":"ContainerStarted","Data":"998918f9893d95de7511d25bc2aa76f51efaa16f4ab671fe16fa66a3cc4b06c2"}
Sep 14 11:08:39.300148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.300101    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b" podStartSLOduration=2.127129703 podStartE2EDuration="14.300087819s" totalImagesPullingTime="12.172958116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:39.297221717 +0000 UTC m=+467.919774931" watchObservedRunningTime="2026-09-14 11:08:39.300087819 +0000 UTC m=+467.922641033"
Sep 14 11:08:39.316846 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.316801    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mpr2j" podStartSLOduration=3.056011099 podStartE2EDuration="9.316787525s" totalImagesPullingTime="6.260776426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:39.314765069 +0000 UTC m=+467.937318284" watchObservedRunningTime="2026-09-14 11:08:39.316787525 +0000 UTC m=+467.939340738"
Sep 14 11:08:39.393693 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.393654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:39.465083 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.465052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:39.983291 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:39.983260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:40.083464 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:40.083434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:42.432066 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:42.432032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:43.523302 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:43.523274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:43.623220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:43.623172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:45.958949 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:45.958916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:46.837450 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:46.837413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:50.277554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:50.277524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sxr4b"
Sep 14 11:08:51.394535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:51.394500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:51.498709 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:51.498672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:08:51.599076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:08:51.599044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:11.519336 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:11.519295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:11.656586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:11.656554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:11.764893 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:11.764855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:11.901952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:11.901909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:12.009745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:12.009707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:14.312783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:14.312744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:14.418541 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:14.418496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:14.761917 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:14.761885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:15.581312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:15.581273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:16.765889 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:16.765853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:26.159151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:26.159114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:29.973584 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:29.973551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:30.905468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:30.905435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:31.010886 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:31.010851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:31.737055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:31.737020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:34.060026 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:34.059987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:39.072536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:39.072500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:39.173609 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:39.173575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:41.075498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:41.075466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:41.190359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:41.190324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:41.283349 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:41.283316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:41.382925 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:41.382896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:47.703415 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.703322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-rcjhz"]
Sep 14 11:09:47.706644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.706627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.709173 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.709151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Sep 14 11:09:47.769761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.769740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/56e26c9f-fbcc-441c-98f3-596287155878-ready\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.769862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.769772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8t6n\" (UniqueName: \"kubernetes.io/projected/56e26c9f-fbcc-441c-98f3-596287155878-kube-api-access-t8t6n\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.769862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.769801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56e26c9f-fbcc-441c-98f3-596287155878-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.769935 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.769868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56e26c9f-fbcc-441c-98f3-596287155878-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870255 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56e26c9f-fbcc-441c-98f3-596287155878-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56e26c9f-fbcc-441c-98f3-596287155878-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/56e26c9f-fbcc-441c-98f3-596287155878-ready\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8t6n\" (UniqueName: \"kubernetes.io/projected/56e26c9f-fbcc-441c-98f3-596287155878-kube-api-access-t8t6n\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56e26c9f-fbcc-441c-98f3-596287155878-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/56e26c9f-fbcc-441c-98f3-596287155878-ready\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.870759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.870741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56e26c9f-fbcc-441c-98f3-596287155878-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:47.887305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:47.887284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8t6n\" (UniqueName: \"kubernetes.io/projected/56e26c9f-fbcc-441c-98f3-596287155878-kube-api-access-t8t6n\") pod \"cni-sysctl-allowlist-ds-rcjhz\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") " pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:48.015710 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.015651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:48.027961 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:09:48.027940    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e26c9f_fbcc_441c_98f3_596287155878.slice/crio-c6036f13a6dc7a7000f9a66ecc62eb7eb2d9bdf9f670dc2bf69a06f8d2a2f2ed WatchSource:0}: Error finding container c6036f13a6dc7a7000f9a66ecc62eb7eb2d9bdf9f670dc2bf69a06f8d2a2f2ed: Status 404 returned error can't find the container with id c6036f13a6dc7a7000f9a66ecc62eb7eb2d9bdf9f670dc2bf69a06f8d2a2f2ed
Sep 14 11:09:48.639981 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.639942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" event={"ID":"56e26c9f-fbcc-441c-98f3-596287155878","Type":"ContainerStarted","Data":"5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba"}
Sep 14 11:09:48.640161 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.640023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" event={"ID":"56e26c9f-fbcc-441c-98f3-596287155878","Type":"ContainerStarted","Data":"c6036f13a6dc7a7000f9a66ecc62eb7eb2d9bdf9f670dc2bf69a06f8d2a2f2ed"}
Sep 14 11:09:48.640209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.640177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:48.685621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.685575    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" podStartSLOduration=1.685561781 podStartE2EDuration="1.685561781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:48.683747433 +0000 UTC m=+537.306300646" watchObservedRunningTime="2026-09-14 11:09:48.685561781 +0000 UTC m=+537.308114994"
Sep 14 11:09:48.771507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.771475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:48.891135 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:48.891063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:49.047268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.047241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:49.203731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.203651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:49.331321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.331290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:49.569103 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.569018    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:09:49.572511 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.572493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.574843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.574822    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 11:09:49.582954 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.582932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:09:49.614822 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.614799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:09:49.656251 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.656226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:09:49.686279 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.686249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08e1dfa2-16bc-4873-88d0-d64c40192fd2-config-file\") pod \"limitador-limitador-84994f7ffc-gxd6k\" (UID: \"08e1dfa2-16bc-4873-88d0-d64c40192fd2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.686428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.686314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjrh\" (UniqueName: \"kubernetes.io/projected/08e1dfa2-16bc-4873-88d0-d64c40192fd2-kube-api-access-7mjrh\") pod \"limitador-limitador-84994f7ffc-gxd6k\" (UID: \"08e1dfa2-16bc-4873-88d0-d64c40192fd2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.787487 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.787439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08e1dfa2-16bc-4873-88d0-d64c40192fd2-config-file\") pod \"limitador-limitador-84994f7ffc-gxd6k\" (UID: \"08e1dfa2-16bc-4873-88d0-d64c40192fd2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.787869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.787539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjrh\" (UniqueName: \"kubernetes.io/projected/08e1dfa2-16bc-4873-88d0-d64c40192fd2-kube-api-access-7mjrh\") pod \"limitador-limitador-84994f7ffc-gxd6k\" (UID: \"08e1dfa2-16bc-4873-88d0-d64c40192fd2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.788174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.788156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08e1dfa2-16bc-4873-88d0-d64c40192fd2-config-file\") pod \"limitador-limitador-84994f7ffc-gxd6k\" (UID: \"08e1dfa2-16bc-4873-88d0-d64c40192fd2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.810062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.810037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjrh\" (UniqueName: \"kubernetes.io/projected/08e1dfa2-16bc-4873-88d0-d64c40192fd2-kube-api-access-7mjrh\") pod \"limitador-limitador-84994f7ffc-gxd6k\" (UID: \"08e1dfa2-16bc-4873-88d0-d64c40192fd2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:49.883886 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:49.883843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:50.011313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:50.011276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:50.110548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:50.110514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:50.166139 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:50.166076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:09:50.168243 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:09:50.168218    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e1dfa2_16bc_4873_88d0_d64c40192fd2.slice/crio-da878100e734107e37e1fe49870ddfaec2dc9dc2f9c86306127918194bfdde8b WatchSource:0}: Error finding container da878100e734107e37e1fe49870ddfaec2dc9dc2f9c86306127918194bfdde8b: Status 404 returned error can't find the container with id da878100e734107e37e1fe49870ddfaec2dc9dc2f9c86306127918194bfdde8b
Sep 14 11:09:50.416857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:50.416776    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-rcjhz"]
Sep 14 11:09:50.652598 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:50.652560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k" event={"ID":"08e1dfa2-16bc-4873-88d0-d64c40192fd2","Type":"ContainerStarted","Data":"da878100e734107e37e1fe49870ddfaec2dc9dc2f9c86306127918194bfdde8b"}
Sep 14 11:09:51.659360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:51.659321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k" event={"ID":"08e1dfa2-16bc-4873-88d0-d64c40192fd2","Type":"ContainerStarted","Data":"5890eca0262a4ce978deb24dbfcdf77feefd07ad20fb408d9c94dfc55ae8a8f9"}
Sep 14 11:09:51.659789 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:51.659426    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" gracePeriod=30
Sep 14 11:09:51.708010 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:51.707965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k" podStartSLOduration=1.8377351339999999 podStartE2EDuration="2.707948694s" totalImagesPullingTime="870.21356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:51.707078148 +0000 UTC m=+540.329631364" watchObservedRunningTime="2026-09-14 11:09:51.707948694 +0000 UTC m=+540.330501907"
Sep 14 11:09:52.184511 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:52.184480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:52.254270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:52.254242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:09:52.665178 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:09:52.665147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:09:59.647806 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:09:59.647761    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:09:59.648799 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:09:59.648762    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:09:59.649600 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:09:59.649576    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:09:59.649658 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:09:59.649611    2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 14 11:10:03.671294 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:03.671263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"
Sep 14 11:10:04.942238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:04.942204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:05.075261 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:05.075223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:05.472970 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:05.472940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:09.647547 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:09.647504    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:10:09.648425 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:09.648376    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:10:09.649273 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:09.649246    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:10:09.649594 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:09.649288    2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 14 11:10:18.267647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:18.267609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:18.791073 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:18.791037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:19.647739 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:19.647697    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:10:19.648845 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:19.648810    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:10:19.649948 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:19.649909    2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 14 11:10:19.650040 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:19.649961    2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 14 11:10:21.744956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.744933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-rcjhz_56e26c9f-fbcc-441c-98f3-596287155878/kube-multus-additional-cni-plugins/0.log"
Sep 14 11:10:21.747049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.747033    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:10:21.823917 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.823889    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-rcjhz_56e26c9f-fbcc-441c-98f3-596287155878/kube-multus-additional-cni-plugins/0.log"
Sep 14 11:10:21.824081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.823928    2563 generic.go:356] "Generic (PLEG): container finished" podID="56e26c9f-fbcc-441c-98f3-596287155878" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba" exitCode=137
Sep 14 11:10:21.824081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.823954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" event={"ID":"56e26c9f-fbcc-441c-98f3-596287155878","Type":"ContainerDied","Data":"5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba"}
Sep 14 11:10:21.824081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.823978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz"
Sep 14 11:10:21.824081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.823988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-rcjhz" event={"ID":"56e26c9f-fbcc-441c-98f3-596287155878","Type":"ContainerDied","Data":"c6036f13a6dc7a7000f9a66ecc62eb7eb2d9bdf9f670dc2bf69a06f8d2a2f2ed"}
Sep 14 11:10:21.824081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.824007    2563 scope.go:117] "RemoveContainer" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba"
Sep 14 11:10:21.833139 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.833118    2563 scope.go:117] "RemoveContainer" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba"
Sep 14 11:10:21.833474 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:10:21.833444    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba\": container with ID starting with 5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba not found: ID does not exist" containerID="5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba"
Sep 14 11:10:21.833614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.833479    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba"} err="failed to get container status \"5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba\": rpc error: code = NotFound desc = could not find container \"5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba\": container with ID starting with 5442ce0e86b4b6c7ef8b1f113a03719f2991fcbaa804a50e0378b74cca70caba not found: ID does not exist"
Sep 14 11:10:21.863368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/56e26c9f-fbcc-441c-98f3-596287155878-ready\") pod \"56e26c9f-fbcc-441c-98f3-596287155878\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") "
Sep 14 11:10:21.863368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56e26c9f-fbcc-441c-98f3-596287155878-tuning-conf-dir\") pod \"56e26c9f-fbcc-441c-98f3-596287155878\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") "
Sep 14 11:10:21.863563 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56e26c9f-fbcc-441c-98f3-596287155878-cni-sysctl-allowlist\") pod \"56e26c9f-fbcc-441c-98f3-596287155878\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") "
Sep 14 11:10:21.863563 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/56e26c9f-fbcc-441c-98f3-596287155878-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "56e26c9f-fbcc-441c-98f3-596287155878" (UID: "56e26c9f-fbcc-441c-98f3-596287155878"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 14 11:10:21.863563 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8t6n\" (UniqueName: \"kubernetes.io/projected/56e26c9f-fbcc-441c-98f3-596287155878-kube-api-access-t8t6n\") pod \"56e26c9f-fbcc-441c-98f3-596287155878\" (UID: \"56e26c9f-fbcc-441c-98f3-596287155878\") "
Sep 14 11:10:21.863684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863614    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e26c9f-fbcc-441c-98f3-596287155878-ready" (OuterVolumeSpecName: "ready") pod "56e26c9f-fbcc-441c-98f3-596287155878" (UID: "56e26c9f-fbcc-441c-98f3-596287155878"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:10:21.863788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e26c9f-fbcc-441c-98f3-596287155878-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "56e26c9f-fbcc-441c-98f3-596287155878" (UID: "56e26c9f-fbcc-441c-98f3-596287155878"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:10:21.863828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863779    2563 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/56e26c9f-fbcc-441c-98f3-596287155878-ready\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:10:21.863828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.863808    2563 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56e26c9f-fbcc-441c-98f3-596287155878-tuning-conf-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:10:21.865598 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.865578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e26c9f-fbcc-441c-98f3-596287155878-kube-api-access-t8t6n" (OuterVolumeSpecName: "kube-api-access-t8t6n") pod "56e26c9f-fbcc-441c-98f3-596287155878" (UID: "56e26c9f-fbcc-441c-98f3-596287155878"). InnerVolumeSpecName "kube-api-access-t8t6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:10:21.964594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.964563    2563 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56e26c9f-fbcc-441c-98f3-596287155878-cni-sysctl-allowlist\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:10:21.964594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:21.964591    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8t6n\" (UniqueName: \"kubernetes.io/projected/56e26c9f-fbcc-441c-98f3-596287155878-kube-api-access-t8t6n\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:10:22.154299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:22.154257    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-rcjhz"]
Sep 14 11:10:22.157675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:22.157647    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-rcjhz"]
Sep 14 11:10:23.919250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:23.919213    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e26c9f-fbcc-441c-98f3-596287155878" path="/var/lib/kubelet/pods/56e26c9f-fbcc-441c-98f3-596287155878/volumes"
Sep 14 11:10:24.779433 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:24.779381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:24.873307 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:24.873279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:24.947323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:24.947290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:25.091708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:25.091627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:28.616160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:28.616124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:28.755229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:28.755198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:10:51.842192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:51.842164    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:10:51.843998 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:10:51.843973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:11:01.577104 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:01.577055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:01.675315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:01.675280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:03.026952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:03.026916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:03.128805 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:03.128766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:03.346217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:03.346117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:03.424472 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:03.424441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:03.956407 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:03.956361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:04.107663 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:04.107628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:04.264963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:04.264871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:11:15.776217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:15.776067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:15.878932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:15.878900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:26.701761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:26.701725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:27.054601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:27.054510    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:11:31.050528 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.050493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"]
Sep 14 11:11:31.050885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.050839    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins"
Sep 14 11:11:31.050885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.050851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins"
Sep 14 11:11:31.050962 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.050916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e26c9f-fbcc-441c-98f3-596287155878" containerName="kube-multus-additional-cni-plugins"
Sep 14 11:11:31.053918 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.053902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.056438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.056418    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 11:11:31.057092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.057073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 11:11:31.057166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.057109    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6r78c\""
Sep 14 11:11:31.057224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.057183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 14 11:11:31.070965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.070944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"]
Sep 14 11:11:31.146927 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.146895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.147077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.146943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75pcp\" (UniqueName: \"kubernetes.io/projected/6986b80e-1be1-44d4-98a4-ce60b159b1a2-kube-api-access-75pcp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.147077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.146969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.147077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.147001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6986b80e-1be1-44d4-98a4-ce60b159b1a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.147224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.147091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.147224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.147116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.247973 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.247939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.247973 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.247976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248200 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248200 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75pcp\" (UniqueName: \"kubernetes.io/projected/6986b80e-1be1-44d4-98a4-ce60b159b1a2-kube-api-access-75pcp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248200 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248200 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6986b80e-1be1-44d4-98a4-ce60b159b1a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248525 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.248582 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.248548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.250357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.250336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6986b80e-1be1-44d4-98a4-ce60b159b1a2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.250587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.250570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6986b80e-1be1-44d4-98a4-ce60b159b1a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.253572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.253551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:31.255677 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.255658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75pcp\" (UniqueName: \"kubernetes.io/projected/6986b80e-1be1-44d4-98a4-ce60b159b1a2-kube-api-access-75pcp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plhf\" (UID: \"6986b80e-1be1-44d4-98a4-ce60b159b1a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.364845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.364809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:31.643777 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.643750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"]
Sep 14 11:11:31.645455 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:11:31.645425    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6986b80e_1be1_44d4_98a4_ce60b159b1a2.slice/crio-ef4b47f53224eef96d929675044a3def94a406b20ad8949de282f528c64a251f WatchSource:0}: Error finding container ef4b47f53224eef96d929675044a3def94a406b20ad8949de282f528c64a251f: Status 404 returned error can't find the container with id ef4b47f53224eef96d929675044a3def94a406b20ad8949de282f528c64a251f
Sep 14 11:11:31.646826 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.646804    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:11:31.966872 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:31.966684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:11:32.203765 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:32.203717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf" event={"ID":"6986b80e-1be1-44d4-98a4-ce60b159b1a2","Type":"ContainerStarted","Data":"ef4b47f53224eef96d929675044a3def94a406b20ad8949de282f528c64a251f"}
Sep 14 11:11:33.211427 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:33.211374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf" event={"ID":"6986b80e-1be1-44d4-98a4-ce60b159b1a2","Type":"ContainerStarted","Data":"d9ee7f38280f60cd3786e44f85d6c7ac24e8bdab1b065bd7eee9a1020d3b4a1e"}
Sep 14 11:11:41.365337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:41.365301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:41.365752 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:41.365365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:41.377707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:41.377679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:41.402122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:41.402066    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf" podStartSLOduration=9.38913348 podStartE2EDuration="10.402050304s" totalImagesPullingTime="1.012916824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:33.234237513 +0000 UTC m=+641.856790727" watchObservedRunningTime="2026-09-14 11:11:41.402050304 +0000 UTC m=+650.024603518"
Sep 14 11:11:42.268904 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:42.268866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plhf"
Sep 14 11:11:45.960357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:45.960308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:11:46.851366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:11:46.851333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:12:01.054879 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:01.054843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:12:02.264858 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:02.264820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:12:06.959091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:06.959057    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"]
Sep 14 11:12:06.964174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:06.964148    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:06.966565 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:06.966534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 14 11:12:06.975152 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:06.975126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"]
Sep 14 11:12:06.991780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:06.991741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:12:07.043176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.043139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d51c855-b610-48f8-95d1-b82cdc8b50a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.043176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.043182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.043454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.043205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.043454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.043249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.043454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.043304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh28\" (UniqueName: \"kubernetes.io/projected/2d51c855-b610-48f8-95d1-b82cdc8b50a8-kube-api-access-mkh28\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.043454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.043336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144325 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh28\" (UniqueName: \"kubernetes.io/projected/2d51c855-b610-48f8-95d1-b82cdc8b50a8-kube-api-access-mkh28\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144325 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d51c855-b610-48f8-95d1-b82cdc8b50a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.144948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.145064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.144960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.146761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.146739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d51c855-b610-48f8-95d1-b82cdc8b50a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.146938 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.146921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d51c855-b610-48f8-95d1-b82cdc8b50a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.152728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.152703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh28\" (UniqueName: \"kubernetes.io/projected/2d51c855-b610-48f8-95d1-b82cdc8b50a8-kube-api-access-mkh28\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r\" (UID: \"2d51c855-b610-48f8-95d1-b82cdc8b50a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.275887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.275792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:07.554818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:07.554781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"]
Sep 14 11:12:07.556631 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:12:07.556601    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d51c855_b610_48f8_95d1_b82cdc8b50a8.slice/crio-56e6d3c291edc0a46ae0f14240299e22f10437a570567301a0779390d6c6898d WatchSource:0}: Error finding container 56e6d3c291edc0a46ae0f14240299e22f10437a570567301a0779390d6c6898d: Status 404 returned error can't find the container with id 56e6d3c291edc0a46ae0f14240299e22f10437a570567301a0779390d6c6898d
Sep 14 11:12:08.403076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:08.403034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r" event={"ID":"2d51c855-b610-48f8-95d1-b82cdc8b50a8","Type":"ContainerStarted","Data":"39e3fc669d3a5740cc52eb605c0d443fcf997d1976316da8e4236d2a62aae098"}
Sep 14 11:12:08.403498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:08.403099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r" event={"ID":"2d51c855-b610-48f8-95d1-b82cdc8b50a8","Type":"ContainerStarted","Data":"56e6d3c291edc0a46ae0f14240299e22f10437a570567301a0779390d6c6898d"}
Sep 14 11:12:08.462609 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:08.462545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r" podStartSLOduration=2.19897466 podStartE2EDuration="2.462525323s" totalImagesPullingTime="263.550663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:12:08.432951584 +0000 UTC m=+677.055504800" watchObservedRunningTime="2026-09-14 11:12:08.462525323 +0000 UTC m=+677.085078535"
Sep 14 11:12:08.463065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:08.463044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:12:17.275963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:17.275925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:17.276453 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:17.275995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:17.288538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:17.288510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:17.461148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:17.461115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r"
Sep 14 11:12:43.797369 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:43.797335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:12:45.451529 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:12:45.451438    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:13:18.426429 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:13:18.426373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:13:20.456737 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:13:20.456700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:14:49.118049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:14:49.118015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:14:58.295424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:14:58.295374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:02.571728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:02.571690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:06.853748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:06.853710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:20.491243 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:20.491196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:20.589248 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:20.589214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:21.678034 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:21.677976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:21.768592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:21.768552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:36.401997 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:36.401949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:36.503173 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:36.503137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:36.814861 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:36.814770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:36.935462 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:36.935417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.086277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.086169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.250823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.250780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.357277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.357226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.466457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.466421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.565349 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.565315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.659827 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.659725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:37.754880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:37.754840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:38.371332 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:38.371292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:38.533213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:38.533179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:38.956801 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:38.956763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:38.985353 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:38.985318    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7947-f7e6cb"]
Sep 14 11:15:38.988914 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:38.988896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:38.998080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:38.998054    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5kv55\""
Sep 14 11:15:39.019971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:39.019937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7947-f7e6cb"]
Sep 14 11:15:39.069001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:39.068966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6zn\" (UniqueName: \"kubernetes.io/projected/0b782b1d-e4c7-47fa-8b54-eba9d088a95e-kube-api-access-8m6zn\") pod \"test-curl-7947-f7e6cb\" (UID: \"0b782b1d-e4c7-47fa-8b54-eba9d088a95e\") " pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:39.169841 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:39.169799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6zn\" (UniqueName: \"kubernetes.io/projected/0b782b1d-e4c7-47fa-8b54-eba9d088a95e-kube-api-access-8m6zn\") pod \"test-curl-7947-f7e6cb\" (UID: \"0b782b1d-e4c7-47fa-8b54-eba9d088a95e\") " pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:39.203384 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:39.203350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6zn\" (UniqueName: \"kubernetes.io/projected/0b782b1d-e4c7-47fa-8b54-eba9d088a95e-kube-api-access-8m6zn\") pod \"test-curl-7947-f7e6cb\" (UID: \"0b782b1d-e4c7-47fa-8b54-eba9d088a95e\") " pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:39.317690 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:39.317581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:39.599100 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:39.599010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7947-f7e6cb"]
Sep 14 11:15:39.602191 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:39.602166    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b782b1d_e4c7_47fa_8b54_eba9d088a95e.slice/crio-94407abffe577cf9affa1497776f031735ccdd91a487497e6a4da20b87ff9c13 WatchSource:0}: Error finding container 94407abffe577cf9affa1497776f031735ccdd91a487497e6a4da20b87ff9c13: Status 404 returned error can't find the container with id 94407abffe577cf9affa1497776f031735ccdd91a487497e6a4da20b87ff9c13
Sep 14 11:15:40.162383 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:40.162348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:15:40.174566 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:40.174531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:40.320635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:40.320595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:40.595271 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:40.595176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7947-f7e6cb" event={"ID":"0b782b1d-e4c7-47fa-8b54-eba9d088a95e","Type":"ContainerStarted","Data":"94407abffe577cf9affa1497776f031735ccdd91a487497e6a4da20b87ff9c13"}
Sep 14 11:15:41.622691 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:41.622642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:41.740365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:41.740315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:41.857668 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:41.857624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:41.953962 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:41.953921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:42.608121 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:42.608083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7947-f7e6cb" event={"ID":"0b782b1d-e4c7-47fa-8b54-eba9d088a95e","Type":"ContainerStarted","Data":"1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d"}
Sep 14 11:15:42.635862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:42.635802    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7947-f7e6cb" podStartSLOduration=2.681960988 podStartE2EDuration="4.635784931s" totalImagesPullingTime="1.953823943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:42.633892491 +0000 UTC m=+891.256445707" watchObservedRunningTime="2026-09-14 11:15:42.635784931 +0000 UTC m=+891.258338144"
Sep 14 11:15:43.008717 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.008625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:43.109638 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.109606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:43.261696 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.261607    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7947-f7e6cb"]
Sep 14 11:15:43.265671 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.265631    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7947-f7e6cb"]
Sep 14 11:15:43.382777 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.382745    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7947-b08909"]
Sep 14 11:15:43.383168 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.383154    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" containerName="test-curl-7947-f7e6cb"
Sep 14 11:15:43.383216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.383171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" containerName="test-curl-7947-f7e6cb"
Sep 14 11:15:43.383252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.383240    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" containerName="test-curl-7947-f7e6cb"
Sep 14 11:15:43.386559 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.386543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:43.394761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.394738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7947-b08909"]
Sep 14 11:15:43.508582 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.508539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qpg\" (UniqueName: \"kubernetes.io/projected/73492f7b-b5cb-46ae-9ae2-79491620c21a-kube-api-access-f6qpg\") pod \"test-curl-7947-b08909\" (UID: \"73492f7b-b5cb-46ae-9ae2-79491620c21a\") " pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:43.543155 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.543066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:43.609228 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.609198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qpg\" (UniqueName: \"kubernetes.io/projected/73492f7b-b5cb-46ae-9ae2-79491620c21a-kube-api-access-f6qpg\") pod \"test-curl-7947-b08909\" (UID: \"73492f7b-b5cb-46ae-9ae2-79491620c21a\") " pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:43.613971 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:43.613942    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7947-f7e6cb" container="test-curl-7947-f7e6cb"
Sep 14 11:15:43.632325 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.632298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qpg\" (UniqueName: \"kubernetes.io/projected/73492f7b-b5cb-46ae-9ae2-79491620c21a-kube-api-access-f6qpg\") pod \"test-curl-7947-b08909\" (UID: \"73492f7b-b5cb-46ae-9ae2-79491620c21a\") " pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:43.696907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.696863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:43.979176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:43.979122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7947-b08909"]
Sep 14 11:15:43.980538 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:43.980505    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73492f7b_b5cb_46ae_9ae2_79491620c21a.slice/crio-611aafe62d7f51286ce6e12ab98adaa71a4e6428dfc51dfd4eeed98f94a4819b WatchSource:0}: Error finding container 611aafe62d7f51286ce6e12ab98adaa71a4e6428dfc51dfd4eeed98f94a4819b: Status 404 returned error can't find the container with id 611aafe62d7f51286ce6e12ab98adaa71a4e6428dfc51dfd4eeed98f94a4819b
Sep 14 11:15:44.621582 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:44.621538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7947-b08909" event={"ID":"73492f7b-b5cb-46ae-9ae2-79491620c21a","Type":"ContainerStarted","Data":"1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e"}
Sep 14 11:15:44.621794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:44.621598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7947-b08909" event={"ID":"73492f7b-b5cb-46ae-9ae2-79491620c21a","Type":"ContainerStarted","Data":"611aafe62d7f51286ce6e12ab98adaa71a4e6428dfc51dfd4eeed98f94a4819b"}
Sep 14 11:15:44.621794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:44.621611    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7947-f7e6cb" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" containerName="test-curl-7947-f7e6cb" containerID="cri-o://1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d" gracePeriod=2
Sep 14 11:15:44.644289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:44.644226    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7947-b08909" podStartSLOduration=1.644203132 podStartE2EDuration="1.644203132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:44.64265554 +0000 UTC m=+893.265208777" watchObservedRunningTime="2026-09-14 11:15:44.644203132 +0000 UTC m=+893.266756348"
Sep 14 11:15:44.685768 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:44.685733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:44.785289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:44.785251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:45.000379 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.000266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:45.254944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.254856    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7947-b08909"]
Sep 14 11:15:45.262114 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.262083    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7947-b08909"]
Sep 14 11:15:45.392540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.392505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:45.519613 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.519504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:45.597137 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.597103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"]
Sep 14 11:15:45.597548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.597533    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73492f7b-b5cb-46ae-9ae2-79491620c21a" containerName="test-curl-7947-b08909"
Sep 14 11:15:45.597598 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.597550    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73492f7b-b5cb-46ae-9ae2-79491620c21a" containerName="test-curl-7947-b08909"
Sep 14 11:15:45.597632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.597625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73492f7b-b5cb-46ae-9ae2-79491620c21a" containerName="test-curl-7947-b08909"
Sep 14 11:15:45.600794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.600777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.609046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.609023    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6f3d0f\""
Sep 14 11:15:45.609382 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.609362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6f3d0f-dockercfg-rnfgd\""
Sep 14 11:15:45.626781 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:45.626747    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7947-b08909" container="test-curl-7947-b08909"
Sep 14 11:15:45.630694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.630666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"]
Sep 14 11:15:45.683679 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.683640    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"]
Sep 14 11:15:45.687272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.687253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.700605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.700578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"]
Sep 14 11:15:45.728125 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.728090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c8ba12-68b4-4969-ac93-93aff2b49e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.728291 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.728153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsntg\" (UniqueName: \"kubernetes.io/projected/56c8ba12-68b4-4969-ac93-93aff2b49e97-kube-api-access-xsntg\") pod \"payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.829659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.829551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722b86d3-0e85-4f0c-b26b-21d07884e2b6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.829659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.829597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmjr\" (UniqueName: \"kubernetes.io/projected/722b86d3-0e85-4f0c-b26b-21d07884e2b6-kube-api-access-dgmjr\") pod \"payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.830123 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.829721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c8ba12-68b4-4969-ac93-93aff2b49e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.830123 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.829786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsntg\" (UniqueName: \"kubernetes.io/projected/56c8ba12-68b4-4969-ac93-93aff2b49e97-kube-api-access-xsntg\") pod \"payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.830434 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.830383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c8ba12-68b4-4969-ac93-93aff2b49e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.844449 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.844421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsntg\" (UniqueName: \"kubernetes.io/projected/56c8ba12-68b4-4969-ac93-93aff2b49e97-kube-api-access-xsntg\") pod \"payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.905182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.905146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:45.911056 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.911032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:45.930951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.930915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722b86d3-0e85-4f0c-b26b-21d07884e2b6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.931082 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.930964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmjr\" (UniqueName: \"kubernetes.io/projected/722b86d3-0e85-4f0c-b26b-21d07884e2b6-kube-api-access-dgmjr\") pod \"payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.931670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.931649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722b86d3-0e85-4f0c-b26b-21d07884e2b6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.952673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.952638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmjr\" (UniqueName: \"kubernetes.io/projected/722b86d3-0e85-4f0c-b26b-21d07884e2b6-kube-api-access-dgmjr\") pod \"payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:45.998523 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:45.998346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:46.112572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.112539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:46.214689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.214661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"]
Sep 14 11:15:46.215893 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:46.215862    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c8ba12_68b4_4969_ac93_93aff2b49e97.slice/crio-b3c6c75f00071c8b2120803b9e05d43e4e443d51b42466bfb3bc63b52dff17dd WatchSource:0}: Error finding container b3c6c75f00071c8b2120803b9e05d43e4e443d51b42466bfb3bc63b52dff17dd: Status 404 returned error can't find the container with id b3c6c75f00071c8b2120803b9e05d43e4e443d51b42466bfb3bc63b52dff17dd
Sep 14 11:15:46.302698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.302666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"]
Sep 14 11:15:46.303427 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:46.303401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722b86d3_0e85_4f0c_b26b_21d07884e2b6.slice/crio-fdcaaa2e2412180c9cb6a59499bc7a044d353341b99ecc351241f8abf258f1c7 WatchSource:0}: Error finding container fdcaaa2e2412180c9cb6a59499bc7a044d353341b99ecc351241f8abf258f1c7: Status 404 returned error can't find the container with id fdcaaa2e2412180c9cb6a59499bc7a044d353341b99ecc351241f8abf258f1c7
Sep 14 11:15:46.303973 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.303947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:46.470198 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.470099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:46.634330 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.634301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x" event={"ID":"56c8ba12-68b4-4969-ac93-93aff2b49e97","Type":"ContainerStarted","Data":"b3c6c75f00071c8b2120803b9e05d43e4e443d51b42466bfb3bc63b52dff17dd"}
Sep 14 11:15:46.635717 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.635691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw" event={"ID":"722b86d3-0e85-4f0c-b26b-21d07884e2b6","Type":"ContainerStarted","Data":"fdcaaa2e2412180c9cb6a59499bc7a044d353341b99ecc351241f8abf258f1c7"}
Sep 14 11:15:46.635816 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.635778    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7947-b08909" podUID="73492f7b-b5cb-46ae-9ae2-79491620c21a" containerName="test-curl-7947-b08909" containerID="cri-o://1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e" gracePeriod=2
Sep 14 11:15:46.800592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.800561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:46.845079 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.845045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:46.940790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.940739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m6zn\" (UniqueName: \"kubernetes.io/projected/0b782b1d-e4c7-47fa-8b54-eba9d088a95e-kube-api-access-8m6zn\") pod \"0b782b1d-e4c7-47fa-8b54-eba9d088a95e\" (UID: \"0b782b1d-e4c7-47fa-8b54-eba9d088a95e\") "
Sep 14 11:15:46.943521 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:46.943479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b782b1d-e4c7-47fa-8b54-eba9d088a95e-kube-api-access-8m6zn" (OuterVolumeSpecName: "kube-api-access-8m6zn") pod "0b782b1d-e4c7-47fa-8b54-eba9d088a95e" (UID: "0b782b1d-e4c7-47fa-8b54-eba9d088a95e"). InnerVolumeSpecName "kube-api-access-8m6zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:15:47.042692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.042601    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6zn\" (UniqueName: \"kubernetes.io/projected/0b782b1d-e4c7-47fa-8b54-eba9d088a95e-kube-api-access-8m6zn\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:15:47.158088 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.158052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:47.308502 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.308378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:47.583169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.583137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:47.648453 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.648427    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" containerID="1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d" exitCode=137
Sep 14 11:15:47.648784 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.648479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:47.648784 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.648526    2563 scope.go:117] "RemoveContainer" containerID="1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d"
Sep 14 11:15:47.657592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.657536    2563 scope.go:117] "RemoveContainer" containerID="1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d"
Sep 14 11:15:47.657915 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:47.657889    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d\": container with ID starting with 1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d not found: ID does not exist" containerID="1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d"
Sep 14 11:15:47.657986 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.657930    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d"} err="failed to get container status \"1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d\": rpc error: code = NotFound desc = could not find container \"1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d\": container with ID starting with 1e633a89ce290d6477d07b51467b09d4b1d93d01d8f3d76d3fa5061a66f7790d not found: ID does not exist"
Sep 14 11:15:47.673792 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:47.673755    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7947-f7e6cb\" is forbidden: User \"system:node:ip-10-0-143-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-154.ec2.internal' and this object" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:47.684529 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.684493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"]
Sep 14 11:15:47.689488 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.689459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:47.695462 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:47.695435    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7947-f7e6cb\" is forbidden: User \"system:node:ip-10-0-143-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-154.ec2.internal' and this object" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" pod="opendatahub/test-curl-7947-f7e6cb"
Sep 14 11:15:47.695613 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.695595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6d272c\""
Sep 14 11:15:47.698181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.698155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:47.717487 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.717449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"]
Sep 14 11:15:47.850676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.850646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b59m\" (UniqueName: \"kubernetes.io/projected/6a908caf-391a-4a14-b1a2-d84a0d99afa8-kube-api-access-6b59m\") pod \"payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:47.851060 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.850717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a908caf-391a-4a14-b1a2-d84a0d99afa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:47.919063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.919031    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b782b1d-e4c7-47fa-8b54-eba9d088a95e" path="/var/lib/kubelet/pods/0b782b1d-e4c7-47fa-8b54-eba9d088a95e/volumes"
Sep 14 11:15:47.951873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.951833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a908caf-391a-4a14-b1a2-d84a0d99afa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:47.952036 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.951934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b59m\" (UniqueName: \"kubernetes.io/projected/6a908caf-391a-4a14-b1a2-d84a0d99afa8-kube-api-access-6b59m\") pod \"payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:47.952594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.952574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a908caf-391a-4a14-b1a2-d84a0d99afa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:47.988495 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:47.988462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b59m\" (UniqueName: \"kubernetes.io/projected/6a908caf-391a-4a14-b1a2-d84a0d99afa8-kube-api-access-6b59m\") pod \"payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:48.000265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.000228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:48.293594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.293505    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"]
Sep 14 11:15:48.297645 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:48.297606    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a908caf_391a_4a14_b1a2_d84a0d99afa8.slice/crio-6b41fb479158c3b16df92dcd9e2902664c1eb6ff4d76b9043bbf6d76114c1c34 WatchSource:0}: Error finding container 6b41fb479158c3b16df92dcd9e2902664c1eb6ff4d76b9043bbf6d76114c1c34: Status 404 returned error can't find the container with id 6b41fb479158c3b16df92dcd9e2902664c1eb6ff4d76b9043bbf6d76114c1c34
Sep 14 11:15:48.539485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.539438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:48.657190 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.657149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw" event={"ID":"722b86d3-0e85-4f0c-b26b-21d07884e2b6","Type":"ContainerStarted","Data":"1c8895c82e87fc68f69999368831c1ab40bfe31f86367e20e11e93c238efaeea"}
Sep 14 11:15:48.657362 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.657230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:15:48.660129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.660102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x" event={"ID":"56c8ba12-68b4-4969-ac93-93aff2b49e97","Type":"ContainerStarted","Data":"06640bc5ba4f399c5fb0d0a1b0a7c7e9d06cefd19a5ef6bd724469d4790090c1"}
Sep 14 11:15:48.660299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.660153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:48.662065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.662042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth" event={"ID":"6a908caf-391a-4a14-b1a2-d84a0d99afa8","Type":"ContainerStarted","Data":"372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2"}
Sep 14 11:15:48.662194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.662085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth" event={"ID":"6a908caf-391a-4a14-b1a2-d84a0d99afa8","Type":"ContainerStarted","Data":"6b41fb479158c3b16df92dcd9e2902664c1eb6ff4d76b9043bbf6d76114c1c34"}
Sep 14 11:15:48.662194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.662184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:48.700337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.700209    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw" podStartSLOduration=2.431676541 podStartE2EDuration="3.700186337s" totalImagesPullingTime="1.268509796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:48.698244989 +0000 UTC m=+897.320798195" watchObservedRunningTime="2026-09-14 11:15:48.700186337 +0000 UTC m=+897.322739553"
Sep 14 11:15:48.737679 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.737608    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x" podStartSLOduration=2.379598223 podStartE2EDuration="3.737582915s" totalImagesPullingTime="1.357984692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:48.734057418 +0000 UTC m=+897.356610632" watchObservedRunningTime="2026-09-14 11:15:48.737582915 +0000 UTC m=+897.360136132"
Sep 14 11:15:48.795435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.794891    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth" podStartSLOduration=1.794862221 podStartE2EDuration="1.794862221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:48.789762055 +0000 UTC m=+897.412315268" watchObservedRunningTime="2026-09-14 11:15:48.794862221 +0000 UTC m=+897.417415437"
Sep 14 11:15:48.806173 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.806146    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:48.861166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.861130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6qpg\" (UniqueName: \"kubernetes.io/projected/73492f7b-b5cb-46ae-9ae2-79491620c21a-kube-api-access-f6qpg\") pod \"73492f7b-b5cb-46ae-9ae2-79491620c21a\" (UID: \"73492f7b-b5cb-46ae-9ae2-79491620c21a\") "
Sep 14 11:15:48.863537 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.863506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73492f7b-b5cb-46ae-9ae2-79491620c21a-kube-api-access-f6qpg" (OuterVolumeSpecName: "kube-api-access-f6qpg") pod "73492f7b-b5cb-46ae-9ae2-79491620c21a" (UID: "73492f7b-b5cb-46ae-9ae2-79491620c21a"). InnerVolumeSpecName "kube-api-access-f6qpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:15:48.962006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:48.961906    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6qpg\" (UniqueName: \"kubernetes.io/projected/73492f7b-b5cb-46ae-9ae2-79491620c21a-kube-api-access-f6qpg\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:15:49.253537 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.253429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:49.336303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.336262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:49.670947 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.670909    2563 generic.go:356] "Generic (PLEG): container finished" podID="73492f7b-b5cb-46ae-9ae2-79491620c21a" containerID="1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e" exitCode=137
Sep 14 11:15:49.671146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.670958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:49.671146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.670986    2563 scope.go:117] "RemoveContainer" containerID="1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e"
Sep 14 11:15:49.682824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.682797    2563 scope.go:117] "RemoveContainer" containerID="1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e"
Sep 14 11:15:49.683205 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:49.683176    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e\": container with ID starting with 1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e not found: ID does not exist" containerID="1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e"
Sep 14 11:15:49.683278 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.683216    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e"} err="failed to get container status \"1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e\": rpc error: code = NotFound desc = could not find container \"1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e\": container with ID starting with 1cf592b385f852c9d32442db130200b812ab8434580df6b2abe89e267b6a122e not found: ID does not exist"
Sep 14 11:15:49.697301 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:15:49.697262    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7947-b08909\" is forbidden: User \"system:node:ip-10-0-143-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-154.ec2.internal' and this object" podUID="73492f7b-b5cb-46ae-9ae2-79491620c21a" pod="opendatahub/test-curl-7947-b08909"
Sep 14 11:15:49.918938 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:49.918902    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73492f7b-b5cb-46ae-9ae2-79491620c21a" path="/var/lib/kubelet/pods/73492f7b-b5cb-46ae-9ae2-79491620c21a/volumes"
Sep 14 11:15:50.137565 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:50.137522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:50.355427 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:50.355368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:51.407076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.407035    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"]
Sep 14 11:15:51.412524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.412498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.414936 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.414898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-27a1a2-serving-cert\""
Sep 14 11:15:51.415195 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.415180    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:15:51.416072 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.416052    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:15:51.416347 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.416327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\""
Sep 14 11:15:51.416469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.416345    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:15:51.428581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.428552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"]
Sep 14 11:15:51.454625 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.454594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:51.487741 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.487707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-maas-api-tls\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.487933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.487762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c82cf5c-d280-4a02-83f0-6c89bb045dec-openshift-service-ca\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.487933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.487808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8dx\" (UniqueName: \"kubernetes.io/projected/8c82cf5c-d280-4a02-83f0-6c89bb045dec-kube-api-access-dk8dx\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.487933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.487837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-metrics-certs\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.588325 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.588288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-maas-api-tls\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.588505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.588344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c82cf5c-d280-4a02-83f0-6c89bb045dec-openshift-service-ca\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.588505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.588381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8dx\" (UniqueName: \"kubernetes.io/projected/8c82cf5c-d280-4a02-83f0-6c89bb045dec-kube-api-access-dk8dx\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.588505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.588433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-metrics-certs\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.589128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.589104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c82cf5c-d280-4a02-83f0-6c89bb045dec-openshift-service-ca\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.590877 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.590856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-metrics-certs\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.590948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.590912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-maas-api-tls\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.595052 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.595026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:51.600988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.600963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8dx\" (UniqueName: \"kubernetes.io/projected/8c82cf5c-d280-4a02-83f0-6c89bb045dec-kube-api-access-dk8dx\") pod \"maas-api-e2e-autores-27a1a2-64b7b8b864-6k265\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.704516 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.704413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:51.723416 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.723356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:51.882673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.882648    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:15:51.886421 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:51.886374    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:15:52.008354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.008325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"]
Sep 14 11:15:52.010060 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:52.010036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c82cf5c_d280_4a02_83f0_6c89bb045dec.slice/crio-61f12e08ec993bb10157495f791530d82893488438c2446f37d4e4e9c4245084 WatchSource:0}: Error finding container 61f12e08ec993bb10157495f791530d82893488438c2446f37d4e4e9c4245084: Status 404 returned error can't find the container with id 61f12e08ec993bb10157495f791530d82893488438c2446f37d4e4e9c4245084
Sep 14 11:15:52.196963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.196930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:52.296895 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.296802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:52.351355 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.351314    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"]
Sep 14 11:15:52.356196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.356170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.359024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.358999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-27a1a2\""
Sep 14 11:15:52.368140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.368111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"]
Sep 14 11:15:52.395786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.395748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a142b-3d10-4264-94b4-54c258e8d4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.395786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.395791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgdg\" (UniqueName: \"kubernetes.io/projected/da6a142b-3d10-4264-94b4-54c258e8d4b0-kube-api-access-7vgdg\") pod \"payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.496982 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.496946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a142b-3d10-4264-94b4-54c258e8d4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.496982 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.496985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgdg\" (UniqueName: \"kubernetes.io/projected/da6a142b-3d10-4264-94b4-54c258e8d4b0-kube-api-access-7vgdg\") pod \"payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.497703 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.497683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a142b-3d10-4264-94b4-54c258e8d4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.516800 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.516775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgdg\" (UniqueName: \"kubernetes.io/projected/da6a142b-3d10-4264-94b4-54c258e8d4b0-kube-api-access-7vgdg\") pod \"payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.670040 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.669967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:52.698202 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.698163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265" event={"ID":"8c82cf5c-d280-4a02-83f0-6c89bb045dec","Type":"ContainerStarted","Data":"61f12e08ec993bb10157495f791530d82893488438c2446f37d4e4e9c4245084"}
Sep 14 11:15:52.785885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.785852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:52.961187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:52.961151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"]
Sep 14 11:15:52.962579 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:15:52.962552    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6a142b_3d10_4264_94b4_54c258e8d4b0.slice/crio-b38fbf899cc0bcf66df6e10ff6c1524575a5469c7a9e785a16217ef544a1f4e2 WatchSource:0}: Error finding container b38fbf899cc0bcf66df6e10ff6c1524575a5469c7a9e785a16217ef544a1f4e2: Status 404 returned error can't find the container with id b38fbf899cc0bcf66df6e10ff6c1524575a5469c7a9e785a16217ef544a1f4e2
Sep 14 11:15:53.476092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:53.476037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:15:53.706022 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:53.705981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s" event={"ID":"da6a142b-3d10-4264-94b4-54c258e8d4b0","Type":"ContainerStarted","Data":"4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823"}
Sep 14 11:15:53.706504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:53.706046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s" event={"ID":"da6a142b-3d10-4264-94b4-54c258e8d4b0","Type":"ContainerStarted","Data":"b38fbf899cc0bcf66df6e10ff6c1524575a5469c7a9e785a16217ef544a1f4e2"}
Sep 14 11:15:53.706504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:53.706123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:15:53.731545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:53.731437    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s" podStartSLOduration=1.731413498 podStartE2EDuration="1.731413498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:53.729927219 +0000 UTC m=+902.352480447" watchObservedRunningTime="2026-09-14 11:15:53.731413498 +0000 UTC m=+902.353966709"
Sep 14 11:15:53.831264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:53.831223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:54.713193 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:54.713159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265" event={"ID":"8c82cf5c-d280-4a02-83f0-6c89bb045dec","Type":"ContainerStarted","Data":"0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895"}
Sep 14 11:15:54.722411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:54.722357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:54.741442 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:54.741365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265" podStartSLOduration=1.245267218 podStartE2EDuration="3.741347784s" totalImagesPullingTime="2.496080566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:54.738800186 +0000 UTC m=+903.361353399" watchObservedRunningTime="2026-09-14 11:15:54.741347784 +0000 UTC m=+903.363900998"
Sep 14 11:15:54.819034 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:54.818938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:55.194911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.194882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:55.278414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.278367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:55.391926 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.391885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:55.490517 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.490410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:55.676963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.676925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:55.719053 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.719018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:15:55.978671 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:55.978628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:56.170446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:56.170408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:56.491872 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:56.491826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:57.843572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:57.843514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:58.576552 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:58.576512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:59.101683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:59.101640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:15:59.673236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:59.673207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:15:59.673410 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:59.673275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:15:59.673410 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:15:59.673307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:16:00.140492 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:00.140449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:00.536983 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:00.536896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:00.654181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:00.654148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:00.852345 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:00.852315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:01.056275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:01.056235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:01.554725 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:01.554693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:01.730807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:01.730773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:16:01.766255 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:01.766172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:03.644931 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:03.644896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:04.664170 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:04.664130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:04.715787 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:04.715756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:16:05.938699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:05.938659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:06.181215 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:06.181171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:06.519740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:06.519703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:06.948698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:06.948658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:07.201624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:07.201525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:07.391969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:07.391922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:08.055705 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:08.055669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:08.290875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:08.290836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:09.033991 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.033950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:09.118645 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.118609    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"]
Sep 14 11:16:09.125944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.125915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.130592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.130568    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"kube-root-ca.crt\""
Sep 14 11:16:09.130592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.130583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"simulator-w4-de0f7a-kserve-self-signed-certs\""
Sep 14 11:16:09.130790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.130587    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"default-dockercfg-556jc\""
Sep 14 11:16:09.130790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.130684    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"openshift-service-ca.crt\""
Sep 14 11:16:09.203820 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.203788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"]
Sep 14 11:16:09.219746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.219710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:09.257512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.257460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-dshm\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.257512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.257520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tls-certs\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.257748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.257594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-home\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.257748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.257679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6n78\" (UniqueName: \"kubernetes.io/projected/b3f2b565-08d1-4a1c-84de-b1db06ca174c-kube-api-access-n6n78\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.257824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.257766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-model-cache\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.257824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.257789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tmp-dir\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.358900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.358852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6n78\" (UniqueName: \"kubernetes.io/projected/b3f2b565-08d1-4a1c-84de-b1db06ca174c-kube-api-access-n6n78\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.358953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-model-cache\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.358985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tmp-dir\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.359038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-dshm\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.359066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tls-certs\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.359089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-home\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359503 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.359475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-model-cache\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.359517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tmp-dir\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.359591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.359543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-home\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.361403 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.361374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-dshm\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.361710 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.361687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tls-certs\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.371067 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.371035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6n78\" (UniqueName: \"kubernetes.io/projected/b3f2b565-08d1-4a1c-84de-b1db06ca174c-kube-api-access-n6n78\") pod \"simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") " pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.435147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.435113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:09.724807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.724779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"]
Sep 14 11:16:09.726759 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:16:09.726731    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3f2b565_08d1_4a1c_84de_b1db06ca174c.slice/crio-5e70214ecfb1baa8193b4b724140d6bfa4246c50b2af6b4f261333409ad2a55d WatchSource:0}: Error finding container 5e70214ecfb1baa8193b4b724140d6bfa4246c50b2af6b4f261333409ad2a55d: Status 404 returned error can't find the container with id 5e70214ecfb1baa8193b4b724140d6bfa4246c50b2af6b4f261333409ad2a55d
Sep 14 11:16:09.803582 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.803541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz" event={"ID":"b3f2b565-08d1-4a1c-84de-b1db06ca174c","Type":"ContainerStarted","Data":"8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6"}
Sep 14 11:16:09.803759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.803594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz" event={"ID":"b3f2b565-08d1-4a1c-84de-b1db06ca174c","Type":"ContainerStarted","Data":"5e70214ecfb1baa8193b4b724140d6bfa4246c50b2af6b4f261333409ad2a55d"}
Sep 14 11:16:09.839375 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.839312    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz" podStartSLOduration=0.839292451 podStartE2EDuration="839.292451ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:09.836994865 +0000 UTC m=+918.459548079" watchObservedRunningTime="2026-09-14 11:16:09.839292451 +0000 UTC m=+918.461845665"
Sep 14 11:16:09.949604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:09.949566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:10.422486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:10.422451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:10.556623 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:10.556590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:11.225641 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:11.225604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:11.670216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:11.670176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:11.782631 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:11.782598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:16:12.426680 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:12.426629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:13.303737 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:13.303697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:13.432229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:13.432197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:13.738077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:13.738038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:13.961270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:13.961234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:14.096432 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.096324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:14.267126 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.267091    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"]
Sep 14 11:16:14.274695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.274675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.278016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.277988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39ab34-dockercfg-s4r82\""
Sep 14 11:16:14.278147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.277996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39ab34\""
Sep 14 11:16:14.289797 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.289769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"]
Sep 14 11:16:14.409724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.409682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.409724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.409721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkr2\" (UniqueName: \"kubernetes.io/projected/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-kube-api-access-ffkr2\") pod \"payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.510537 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.510496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.510537 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.510537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkr2\" (UniqueName: \"kubernetes.io/projected/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-kube-api-access-ffkr2\") pod \"payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.511310 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.511289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.520620 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.520586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkr2\" (UniqueName: \"kubernetes.io/projected/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-kube-api-access-ffkr2\") pod \"payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.586065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.586023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:14.706504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.706415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:14.871916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:14.871890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"]
Sep 14 11:16:14.873335 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:16:14.873296    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ccf030_e9b0_4e42_b945_4d935c7b4bc5.slice/crio-079c7defa8948889e49ff723a50525470fd20cee840aae933af88fdd9d1e32f9 WatchSource:0}: Error finding container 079c7defa8948889e49ff723a50525470fd20cee840aae933af88fdd9d1e32f9: Status 404 returned error can't find the container with id 079c7defa8948889e49ff723a50525470fd20cee840aae933af88fdd9d1e32f9
Sep 14 11:16:15.728654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:15.728611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:15.846603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:15.846567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6" event={"ID":"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5","Type":"ContainerStarted","Data":"67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535"}
Sep 14 11:16:15.846797 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:15.846623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6" event={"ID":"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5","Type":"ContainerStarted","Data":"079c7defa8948889e49ff723a50525470fd20cee840aae933af88fdd9d1e32f9"}
Sep 14 11:16:15.846797 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:15.846675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:15.870371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:15.870310    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6" podStartSLOduration=1.870287571 podStartE2EDuration="1.870287571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:15.868967184 +0000 UTC m=+924.491520401" watchObservedRunningTime="2026-09-14 11:16:15.870287571 +0000 UTC m=+924.492840790"
Sep 14 11:16:16.710593 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:16.710558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:16.930359 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:16.930316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:17.846545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:17.846508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:17.976073 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:17.976038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:19.435727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:19.435683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:19.435727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:19.435739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:19.448177 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:19.448143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:19.887732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:19.887702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:16:20.135492 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:20.135448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:20.237211 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:20.237112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:20.338761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:20.338724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:21.886006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:21.885970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:21.988995 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:21.988958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:22.161825 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:22.161728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:22.454275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:22.454178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:22.745843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:22.745753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:23.157977 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:23.157938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:23.267219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:23.267185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:23.366286 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:23.366253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:23.677246 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:23.677211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:24.175898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:24.175858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:24.719173 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:24.719132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:24.854031 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:24.853994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:25.183632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:25.183599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:26.290128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:26.290092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:26.855249 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:26.855215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:16:27.292825 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:27.292737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:28.162757 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:28.162720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:29.080216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:29.080184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:30.224584 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:30.224536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:30.332256 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:30.332211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:30.437429 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:30.437304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:31.047549 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.047511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:31.149871 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.149837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:31.275249 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.275215    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"]
Sep 14 11:16:31.279192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.279168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.281874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.281851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ac71380\""
Sep 14 11:16:31.282001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.281895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ac71380-dockercfg-k2gvr\""
Sep 14 11:16:31.299314 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.299230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"]
Sep 14 11:16:31.369838 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.369793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6fbe84c-f517-4449-bea1-c79757bd85d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.370024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.369902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltpn\" (UniqueName: \"kubernetes.io/projected/b6fbe84c-f517-4449-bea1-c79757bd85d6-kube-api-access-cltpn\") pod \"payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.470697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.470656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6fbe84c-f517-4449-bea1-c79757bd85d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.470887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.470747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltpn\" (UniqueName: \"kubernetes.io/projected/b6fbe84c-f517-4449-bea1-c79757bd85d6-kube-api-access-cltpn\") pod \"payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.471348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.471328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6fbe84c-f517-4449-bea1-c79757bd85d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.483818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.483793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltpn\" (UniqueName: \"kubernetes.io/projected/b6fbe84c-f517-4449-bea1-c79757bd85d6-kube-api-access-cltpn\") pod \"payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.589091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.588996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:31.619536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.619496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:31.787001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.786971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:31.881483 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.881399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"]
Sep 14 11:16:31.882634 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:16:31.882607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fbe84c_f517_4449_bea1_c79757bd85d6.slice/crio-00aa222ab09f316e3dff763317062fb4070da55e2f6b57086cd4a907979c517e WatchSource:0}: Error finding container 00aa222ab09f316e3dff763317062fb4070da55e2f6b57086cd4a907979c517e: Status 404 returned error can't find the container with id 00aa222ab09f316e3dff763317062fb4070da55e2f6b57086cd4a907979c517e
Sep 14 11:16:31.945962 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:31.945934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt" event={"ID":"b6fbe84c-f517-4449-bea1-c79757bd85d6","Type":"ContainerStarted","Data":"00aa222ab09f316e3dff763317062fb4070da55e2f6b57086cd4a907979c517e"}
Sep 14 11:16:32.189501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:32.189417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:16:32.836004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:32.835964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:32.954008 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:32.953969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt" event={"ID":"b6fbe84c-f517-4449-bea1-c79757bd85d6","Type":"ContainerStarted","Data":"3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c"}
Sep 14 11:16:32.954206 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:32.954066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:34.014896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:34.014859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:34.884196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:34.884156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:35.284871 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:35.284782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:35.814028 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:35.813991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:36.604672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:36.604634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:37.586399 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:37.586344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:38.476760 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:38.476723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:43.965983 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:43.965947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:16:43.988686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:43.988622    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt" podStartSLOduration=12.988605415 podStartE2EDuration="12.988605415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:32.986001431 +0000 UTC m=+941.608554646" watchObservedRunningTime="2026-09-14 11:16:43.988605415 +0000 UTC m=+952.611158629"
Sep 14 11:16:48.151311 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:48.151272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:16:51.637915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:51.637876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:53.543647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:53.543609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:54.352503 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:54.352466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:55.342845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:55.342798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:56.340530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:56.340494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:57.169969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:57.169923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:58.118748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:58.118713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:16:59.126725 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:16:59.126695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:00.172784 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:00.172737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:01.021328 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:01.021287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:01.871418 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:01.871350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:02.458670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.458630    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"]
Sep 14 11:17:02.462524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.462499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.466456 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.466430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"default-dockercfg-f2rbx\""
Sep 14 11:17:02.467202 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.467176    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"openshift-service-ca.crt\""
Sep 14 11:17:02.467327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.467218    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"kube-root-ca.crt\""
Sep 14 11:17:02.467327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.467228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"simulator-w0-6f3d0f-kserve-self-signed-certs\""
Sep 14 11:17:02.484579 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.484548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"]
Sep 14 11:17:02.527709 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.527675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:02.566411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.566359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-dshm\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.566411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.566415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tls-certs\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.566632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.566486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-model-cache\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.566632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.566547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tmp-dir\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.566632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.566567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2p9\" (UniqueName: \"kubernetes.io/projected/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-kube-api-access-4w2p9\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.566752 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.566666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-home\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-model-cache\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tmp-dir\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2p9\" (UniqueName: \"kubernetes.io/projected/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-kube-api-access-4w2p9\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-home\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-dshm\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tls-certs\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-model-cache\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tmp-dir\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.668718 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.668662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-home\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.670745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.670717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-dshm\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.670924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.670907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tls-certs\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.677918 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.677890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2p9\" (UniqueName: \"kubernetes.io/projected/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-kube-api-access-4w2p9\") pod \"simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") " pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:02.772275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:02.772169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:03.051102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:03.051017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"]
Sep 14 11:17:03.054211 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:17:03.054179    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf002c3af_b9f8_420a_8a8f_f6b7c05b0e64.slice/crio-25d5e2e661590bb71d3eb48746749a6640deb18539122c9fa7c7e166285649e8 WatchSource:0}: Error finding container 25d5e2e661590bb71d3eb48746749a6640deb18539122c9fa7c7e166285649e8: Status 404 returned error can't find the container with id 25d5e2e661590bb71d3eb48746749a6640deb18539122c9fa7c7e166285649e8
Sep 14 11:17:03.134602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:03.134565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch" event={"ID":"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64","Type":"ContainerStarted","Data":"6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd"}
Sep 14 11:17:03.134786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:03.134614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch" event={"ID":"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64","Type":"ContainerStarted","Data":"25d5e2e661590bb71d3eb48746749a6640deb18539122c9fa7c7e166285649e8"}
Sep 14 11:17:03.171098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:03.171042    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch" podStartSLOduration=1.171024175 podStartE2EDuration="1.171024175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:03.169101052 +0000 UTC m=+971.791654267" watchObservedRunningTime="2026-09-14 11:17:03.171024175 +0000 UTC m=+971.793577388"
Sep 14 11:17:06.365546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:06.365503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.056568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.056529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.096822 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.096785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"]
Sep 14 11:17:07.097146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.097116    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt" podUID="b6fbe84c-f517-4449-bea1-c79757bd85d6" containerName="payload-pre-processing" containerID="cri-o://3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c" gracePeriod=30
Sep 14 11:17:07.189135 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.189098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.305871 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.305835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.358940 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.358911    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:17:07.473947 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.473910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.513677 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.513632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cltpn\" (UniqueName: \"kubernetes.io/projected/b6fbe84c-f517-4449-bea1-c79757bd85d6-kube-api-access-cltpn\") pod \"b6fbe84c-f517-4449-bea1-c79757bd85d6\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") "
Sep 14 11:17:07.513882 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.513707    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6fbe84c-f517-4449-bea1-c79757bd85d6-plugins-config-volume\") pod \"b6fbe84c-f517-4449-bea1-c79757bd85d6\" (UID: \"b6fbe84c-f517-4449-bea1-c79757bd85d6\") "
Sep 14 11:17:07.514078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.514051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fbe84c-f517-4449-bea1-c79757bd85d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6fbe84c-f517-4449-bea1-c79757bd85d6" (UID: "b6fbe84c-f517-4449-bea1-c79757bd85d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:17:07.516019 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.515994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fbe84c-f517-4449-bea1-c79757bd85d6-kube-api-access-cltpn" (OuterVolumeSpecName: "kube-api-access-cltpn") pod "b6fbe84c-f517-4449-bea1-c79757bd85d6" (UID: "b6fbe84c-f517-4449-bea1-c79757bd85d6"). InnerVolumeSpecName "kube-api-access-cltpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:17:07.579673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.579631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.614868 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.614828    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltpn\" (UniqueName: \"kubernetes.io/projected/b6fbe84c-f517-4449-bea1-c79757bd85d6-kube-api-access-cltpn\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:17:07.614868 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.614866    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6fbe84c-f517-4449-bea1-c79757bd85d6-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:17:07.844575 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.844539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:07.872240 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.872142    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:17:07.966078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:07.966041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:08.088701 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.088670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:08.166133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.166043    2563 generic.go:356] "Generic (PLEG): container finished" podID="b6fbe84c-f517-4449-bea1-c79757bd85d6" containerID="3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c" exitCode=0
Sep 14 11:17:08.166133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.166088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"
Sep 14 11:17:08.166315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.166129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt" event={"ID":"b6fbe84c-f517-4449-bea1-c79757bd85d6","Type":"ContainerDied","Data":"3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c"}
Sep 14 11:17:08.166315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.166181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt" event={"ID":"b6fbe84c-f517-4449-bea1-c79757bd85d6","Type":"ContainerDied","Data":"00aa222ab09f316e3dff763317062fb4070da55e2f6b57086cd4a907979c517e"}
Sep 14 11:17:08.166315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.166210    2563 scope.go:117] "RemoveContainer" containerID="3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c"
Sep 14 11:17:08.175990 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.175966    2563 scope.go:117] "RemoveContainer" containerID="3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c"
Sep 14 11:17:08.176303 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:17:08.176284    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c\": container with ID starting with 3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c not found: ID does not exist" containerID="3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c"
Sep 14 11:17:08.176405 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.176310    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c"} err="failed to get container status \"3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c\": rpc error: code = NotFound desc = could not find container \"3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c\": container with ID starting with 3cb57d06b50fa01d4a9b51620ee358e4604e8054c59395d12a1c485fa34aad7c not found: ID does not exist"
Sep 14 11:17:08.195664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.195620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"]
Sep 14 11:17:08.200942 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.200906    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ac71380-8569587574-65hqt"]
Sep 14 11:17:08.220256 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.220217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:08.620312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.620278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:08.808358 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:08.808318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:09.163438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:09.163376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:09.919669 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:09.919635    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6fbe84c-f517-4449-bea1-c79757bd85d6" path="/var/lib/kubelet/pods/b6fbe84c-f517-4449-bea1-c79757bd85d6/volumes"
Sep 14 11:17:10.421890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:10.421848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:11.059074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:11.059036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:11.167190 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:11.167145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:11.496678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:11.496640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:12.488596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:12.488552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:12.773259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:12.773175    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:12.773259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:12.773227    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:12.786487 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:12.786454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:13.211686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:13.211657    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:17:13.402300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:13.402266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:14.156454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.156365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:14.256428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.256376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:14.561014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.560913    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"]
Sep 14 11:17:14.562239 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.562206    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6fbe84c-f517-4449-bea1-c79757bd85d6" containerName="payload-pre-processing"
Sep 14 11:17:14.562434 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.562417    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6fbe84c-f517-4449-bea1-c79757bd85d6" containerName="payload-pre-processing"
Sep 14 11:17:14.562772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.562756    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6fbe84c-f517-4449-bea1-c79757bd85d6" containerName="payload-pre-processing"
Sep 14 11:17:14.568312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.568289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.571617 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.571594    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-27a1a2\"/\"default-dockercfg-rw6pb\""
Sep 14 11:17:14.572156 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.572129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-27a1a2\"/\"openshift-service-ca.crt\""
Sep 14 11:17:14.572418 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.572228    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-27a1a2\"/\"kube-root-ca.crt\""
Sep 14 11:17:14.572780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.572751    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-27a1a2\"/\"autores-model-27a1a2-kserve-self-signed-certs\""
Sep 14 11:17:14.573206 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.573153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:14.574229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.574208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"]
Sep 14 11:17:14.577177 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.577156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-model-cache\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.577267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.577252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g6k\" (UniqueName: \"kubernetes.io/projected/484a937a-e12d-43a7-92c4-a2a42b4538d9-kube-api-access-65g6k\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.577324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.577289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-tmp-dir\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.577367 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.577335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-home\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.577451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.577376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484a937a-e12d-43a7-92c4-a2a42b4538d9-tls-certs\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.577512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.577462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-dshm\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678253 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-tmp-dir\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-home\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484a937a-e12d-43a7-92c4-a2a42b4538d9-tls-certs\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-dshm\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-model-cache\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g6k\" (UniqueName: \"kubernetes.io/projected/484a937a-e12d-43a7-92c4-a2a42b4538d9-kube-api-access-65g6k\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-tmp-dir\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-home\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.678869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.678838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-model-cache\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.680650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.680623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-dshm\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.681018 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.680999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484a937a-e12d-43a7-92c4-a2a42b4538d9-tls-certs\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.686318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.686292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:14.716873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.716833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g6k\" (UniqueName: \"kubernetes.io/projected/484a937a-e12d-43a7-92c4-a2a42b4538d9-kube-api-access-65g6k\") pod \"autores-model-27a1a2-kserve-7887b85887-dmm8b\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") " pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.879703 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.879666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:14.997194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:14.997155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:15.093017 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:15.092983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:15.166221 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:15.166190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"]
Sep 14 11:17:15.168240 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:17:15.168213    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod484a937a_e12d_43a7_92c4_a2a42b4538d9.slice/crio-9a47eca637f75d7c7bc8091c7c690aacd6ddd6fbee4bb278f87fb271093962e1 WatchSource:0}: Error finding container 9a47eca637f75d7c7bc8091c7c690aacd6ddd6fbee4bb278f87fb271093962e1: Status 404 returned error can't find the container with id 9a47eca637f75d7c7bc8091c7c690aacd6ddd6fbee4bb278f87fb271093962e1
Sep 14 11:17:15.214062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:15.214037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b" event={"ID":"484a937a-e12d-43a7-92c4-a2a42b4538d9","Type":"ContainerStarted","Data":"9a47eca637f75d7c7bc8091c7c690aacd6ddd6fbee4bb278f87fb271093962e1"}
Sep 14 11:17:15.625621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:15.625577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:16.221758 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:16.221720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b" event={"ID":"484a937a-e12d-43a7-92c4-a2a42b4538d9","Type":"ContainerStarted","Data":"70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee"}
Sep 14 11:17:16.246518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:16.246457    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b" podStartSLOduration=2.246441255 podStartE2EDuration="2.246441255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:16.245097107 +0000 UTC m=+984.867650321" watchObservedRunningTime="2026-09-14 11:17:16.246441255 +0000 UTC m=+984.868994468"
Sep 14 11:17:16.562000 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:16.561912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:17.559485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:17.559444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:22.712443 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:22.712384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:17:24.880727 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:24.880677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:24.881125 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:24.880744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:24.893602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:24.893571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:25.284513 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:25.284424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:17:37.302403 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:37.302354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:37.404206 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:37.404169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:38.407588 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:38.407540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:39.173381 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:39.173330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:40.133629 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:40.133595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:41.213694 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:41.213653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:42.208839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:42.208807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:43.148635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:43.148600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:44.115039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:44.114996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:45.086306 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:45.086268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:47.512923 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:47.512891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:47.652133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:47.652102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:48.401151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:48.401082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:49.406090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:49.406053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:50.446957 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:50.446918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:51.527597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:51.527564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:52.832953 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:52.832919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:52.928186 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:52.928150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:54.050678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:54.050629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:54.928095 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:54.928057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:55.655837 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:55.655805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:55.730403 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:55.730360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:55.794751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:55.794719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:55.879968 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:55.879931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:55.949175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:55.949078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:56.055460 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:56.055422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:56.413792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:56.413747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:56.515166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:56.515129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:56.848650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:56.848553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:57.085338 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:57.085296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:57.307818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:57.307784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:58.512944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:58.512888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:59.198457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:59.198417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:59.294768 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:59.294733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:17:59.832642 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:17:59.832605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:00.750691 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.750657    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"]
Sep 14 11:18:00.750978 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.750938    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265" podUID="8c82cf5c-d280-4a02-83f0-6c89bb045dec" containerName="maas-api" containerID="cri-o://0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895" gracePeriod=30
Sep 14 11:18:00.776808 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.776771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:18:00.794748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.794703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:00.925896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.925859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"]
Sep 14 11:18:00.926272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.926104    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s" podUID="da6a142b-3d10-4264-94b4-54c258e8d4b0" containerName="payload-pre-processing" containerID="cri-o://4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823" gracePeriod=30
Sep 14 11:18:00.969071 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:00.969029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:01.054586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.054555    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:18:01.059135 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.059101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:01.112980 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.112946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-maas-api-tls\") pod \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") "
Sep 14 11:18:01.113174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.112995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk8dx\" (UniqueName: \"kubernetes.io/projected/8c82cf5c-d280-4a02-83f0-6c89bb045dec-kube-api-access-dk8dx\") pod \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") "
Sep 14 11:18:01.113174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.113067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-metrics-certs\") pod \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") "
Sep 14 11:18:01.113174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.113098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c82cf5c-d280-4a02-83f0-6c89bb045dec-openshift-service-ca\") pod \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\" (UID: \"8c82cf5c-d280-4a02-83f0-6c89bb045dec\") "
Sep 14 11:18:01.113486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.113458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c82cf5c-d280-4a02-83f0-6c89bb045dec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c82cf5c-d280-4a02-83f0-6c89bb045dec" (UID: "8c82cf5c-d280-4a02-83f0-6c89bb045dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:18:01.116598 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.116571    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c82cf5c-d280-4a02-83f0-6c89bb045dec" (UID: "8c82cf5c-d280-4a02-83f0-6c89bb045dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:01.116696 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.116580    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c82cf5c-d280-4a02-83f0-6c89bb045dec-kube-api-access-dk8dx" (OuterVolumeSpecName: "kube-api-access-dk8dx") pod "8c82cf5c-d280-4a02-83f0-6c89bb045dec" (UID: "8c82cf5c-d280-4a02-83f0-6c89bb045dec"). InnerVolumeSpecName "kube-api-access-dk8dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:01.116696 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.116597    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c82cf5c-d280-4a02-83f0-6c89bb045dec" (UID: "8c82cf5c-d280-4a02-83f0-6c89bb045dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:01.184853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.184827    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:18:01.213766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.213736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vgdg\" (UniqueName: \"kubernetes.io/projected/da6a142b-3d10-4264-94b4-54c258e8d4b0-kube-api-access-7vgdg\") pod \"da6a142b-3d10-4264-94b4-54c258e8d4b0\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") "
Sep 14 11:18:01.213949 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.213816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a142b-3d10-4264-94b4-54c258e8d4b0-plugins-config-volume\") pod \"da6a142b-3d10-4264-94b4-54c258e8d4b0\" (UID: \"da6a142b-3d10-4264-94b4-54c258e8d4b0\") "
Sep 14 11:18:01.214043 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.214021    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:01.214043 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.214036    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c82cf5c-d280-4a02-83f0-6c89bb045dec-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:01.214154 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.214046    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c82cf5c-d280-4a02-83f0-6c89bb045dec-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:01.214154 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.214070    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8dx\" (UniqueName: \"kubernetes.io/projected/8c82cf5c-d280-4a02-83f0-6c89bb045dec-kube-api-access-dk8dx\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:01.215129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.214426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6a142b-3d10-4264-94b4-54c258e8d4b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da6a142b-3d10-4264-94b4-54c258e8d4b0" (UID: "da6a142b-3d10-4264-94b4-54c258e8d4b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:18:01.216712 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.216686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6a142b-3d10-4264-94b4-54c258e8d4b0-kube-api-access-7vgdg" (OuterVolumeSpecName: "kube-api-access-7vgdg") pod "da6a142b-3d10-4264-94b4-54c258e8d4b0" (UID: "da6a142b-3d10-4264-94b4-54c258e8d4b0"). InnerVolumeSpecName "kube-api-access-7vgdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:01.298106 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.297990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:01.314594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.314553    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6a142b-3d10-4264-94b4-54c258e8d4b0-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:01.314594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.314586    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgdg\" (UniqueName: \"kubernetes.io/projected/da6a142b-3d10-4264-94b4-54c258e8d4b0-kube-api-access-7vgdg\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:01.443228 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.443185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:01.495037 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.495000    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c82cf5c-d280-4a02-83f0-6c89bb045dec" containerID="0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895" exitCode=0
Sep 14 11:18:01.495213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.495048    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"
Sep 14 11:18:01.495213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.495088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265" event={"ID":"8c82cf5c-d280-4a02-83f0-6c89bb045dec","Type":"ContainerDied","Data":"0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895"}
Sep 14 11:18:01.495213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.495140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265" event={"ID":"8c82cf5c-d280-4a02-83f0-6c89bb045dec","Type":"ContainerDied","Data":"61f12e08ec993bb10157495f791530d82893488438c2446f37d4e4e9c4245084"}
Sep 14 11:18:01.495213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.495170    2563 scope.go:117] "RemoveContainer" containerID="0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895"
Sep 14 11:18:01.496697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.496665    2563 generic.go:356] "Generic (PLEG): container finished" podID="da6a142b-3d10-4264-94b4-54c258e8d4b0" containerID="4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823" exitCode=0
Sep 14 11:18:01.496697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.496696    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"
Sep 14 11:18:01.496858 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.496749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s" event={"ID":"da6a142b-3d10-4264-94b4-54c258e8d4b0","Type":"ContainerDied","Data":"4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823"}
Sep 14 11:18:01.496858 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.496789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s" event={"ID":"da6a142b-3d10-4264-94b4-54c258e8d4b0","Type":"ContainerDied","Data":"b38fbf899cc0bcf66df6e10ff6c1524575a5469c7a9e785a16217ef544a1f4e2"}
Sep 14 11:18:01.504974 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.504950    2563 scope.go:117] "RemoveContainer" containerID="0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895"
Sep 14 11:18:01.505297 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:18:01.505275    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895\": container with ID starting with 0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895 not found: ID does not exist" containerID="0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895"
Sep 14 11:18:01.505374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.505304    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895"} err="failed to get container status \"0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895\": rpc error: code = NotFound desc = could not find container \"0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895\": container with ID starting with 0dafa0a5894c0bb6fd7ae72380ced1fe640274d4303d6defb1bca54abd345895 not found: ID does not exist"
Sep 14 11:18:01.505374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.505321    2563 scope.go:117] "RemoveContainer" containerID="4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823"
Sep 14 11:18:01.514703 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.514683    2563 scope.go:117] "RemoveContainer" containerID="4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823"
Sep 14 11:18:01.514992 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:18:01.514968    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823\": container with ID starting with 4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823 not found: ID does not exist" containerID="4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823"
Sep 14 11:18:01.515071 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.515001    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823"} err="failed to get container status \"4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823\": rpc error: code = NotFound desc = could not find container \"4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823\": container with ID starting with 4fa66665adbd9f4d45ef7c629f7ea92fbc88b87c9cbef3429c48819b3a1a5823 not found: ID does not exist"
Sep 14 11:18:01.539217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.539181    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"]
Sep 14 11:18:01.552775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.552680    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-27a1a2-64b7b8b864-6k265"]
Sep 14 11:18:01.585193 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.585143    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"]
Sep 14 11:18:01.596994 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.596954    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-27a1a2-76575ff5f4-wbm5s"]
Sep 14 11:18:01.786037 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.786001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:01.919625 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.919587    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c82cf5c-d280-4a02-83f0-6c89bb045dec" path="/var/lib/kubelet/pods/8c82cf5c-d280-4a02-83f0-6c89bb045dec/volumes"
Sep 14 11:18:01.919977 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:01.919962    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6a142b-3d10-4264-94b4-54c258e8d4b0" path="/var/lib/kubelet/pods/da6a142b-3d10-4264-94b4-54c258e8d4b0/volumes"
Sep 14 11:18:02.600673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.600630    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"]
Sep 14 11:18:02.601246 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.601225    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c82cf5c-d280-4a02-83f0-6c89bb045dec" containerName="maas-api"
Sep 14 11:18:02.601337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.601249    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c82cf5c-d280-4a02-83f0-6c89bb045dec" containerName="maas-api"
Sep 14 11:18:02.601337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.601270    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6a142b-3d10-4264-94b4-54c258e8d4b0" containerName="payload-pre-processing"
Sep 14 11:18:02.601337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.601280    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6a142b-3d10-4264-94b4-54c258e8d4b0" containerName="payload-pre-processing"
Sep 14 11:18:02.601522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.601431    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6a142b-3d10-4264-94b4-54c258e8d4b0" containerName="payload-pre-processing"
Sep 14 11:18:02.601522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.601450    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c82cf5c-d280-4a02-83f0-6c89bb045dec" containerName="maas-api"
Sep 14 11:18:02.606515 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.606489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.610192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.610162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-88bdd8bf-serving-cert\""
Sep 14 11:18:02.610366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.610275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\""
Sep 14 11:18:02.610527 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.610512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:18:02.611080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.611063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:18:02.611187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.611077    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:18:02.620480 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.620453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"]
Sep 14 11:18:02.623853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.623817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-metrics-certs\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.624005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.623914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.624005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.623947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806adac7-e76e-41be-8651-64f9fade816a-openshift-service-ca\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.624005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.623982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbpm\" (UniqueName: \"kubernetes.io/projected/806adac7-e76e-41be-8651-64f9fade816a-kube-api-access-6mbpm\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.641892 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.641855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:02.723373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.723324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:02.724548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.724519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.724735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.724560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806adac7-e76e-41be-8651-64f9fade816a-openshift-service-ca\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.724735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.724632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbpm\" (UniqueName: \"kubernetes.io/projected/806adac7-e76e-41be-8651-64f9fade816a-kube-api-access-6mbpm\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.724735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.724722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-metrics-certs\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.724919 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:18:02.724805    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-serving-cert: secret "maas-api-e2e-ait-88bdd8bf-serving-cert" not found
Sep 14 11:18:02.724919 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:18:02.724874    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls podName:806adac7-e76e-41be-8651-64f9fade816a nodeName:}" failed. No retries permitted until 2026-09-14 11:18:03.22485442 +0000 UTC m=+1031.847407612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls") pod "maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" (UID: "806adac7-e76e-41be-8651-64f9fade816a") : secret "maas-api-e2e-ait-88bdd8bf-serving-cert" not found
Sep 14 11:18:02.725276 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.725253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806adac7-e76e-41be-8651-64f9fade816a-openshift-service-ca\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.727340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.727319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-metrics-certs\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:02.748536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:02.748498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbpm\" (UniqueName: \"kubernetes.io/projected/806adac7-e76e-41be-8651-64f9fade816a-kube-api-access-6mbpm\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:03.230013 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:03.229977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:03.232456 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:03.232433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls\") pod \"maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:03.518689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:03.518578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:03.853230 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:03.853198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"]
Sep 14 11:18:03.855087 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:18:03.855055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806adac7_e76e_41be_8651_64f9fade816a.slice/crio-9faeb207fb85333967bfd57d5f2740f0533e6634f37e1db26b91d4effc699474 WatchSource:0}: Error finding container 9faeb207fb85333967bfd57d5f2740f0533e6634f37e1db26b91d4effc699474: Status 404 returned error can't find the container with id 9faeb207fb85333967bfd57d5f2740f0533e6634f37e1db26b91d4effc699474
Sep 14 11:18:03.856606 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:03.856585    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:18:04.381401 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.381341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:04.477312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.477211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:04.518807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.518771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" event={"ID":"806adac7-e76e-41be-8651-64f9fade816a","Type":"ContainerStarted","Data":"25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4"}
Sep 14 11:18:04.518984 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.518857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" event={"ID":"806adac7-e76e-41be-8651-64f9fade816a","Type":"ContainerStarted","Data":"9faeb207fb85333967bfd57d5f2740f0533e6634f37e1db26b91d4effc699474"}
Sep 14 11:18:04.518984 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.518878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:04.545360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.545307    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" podStartSLOduration=2.194598512 podStartE2EDuration="2.5452908s" totalImagesPullingTime="350.692288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:04.544914998 +0000 UTC m=+1033.167468218" watchObservedRunningTime="2026-09-14 11:18:04.5452908 +0000 UTC m=+1033.167844015"
Sep 14 11:18:04.712954 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:04.712914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:05.680238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:05.680199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:06.638844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:06.638802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:06.745211 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:06.745168    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"]
Sep 14 11:18:06.841746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:06.841714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:07.679602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:07.679559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:08.450794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:08.450762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:09.413383 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:09.413332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:10.530223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:10.530194    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:17.571321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:17.571286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:21.764932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:21.764894    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b" podUID="484a937a-e12d-43a7-92c4-a2a42b4538d9" containerName="main" containerID="cri-o://70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee" gracePeriod=45
Sep 14 11:18:22.023780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.023715    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:18:22.106735 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.106704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65g6k\" (UniqueName: \"kubernetes.io/projected/484a937a-e12d-43a7-92c4-a2a42b4538d9-kube-api-access-65g6k\") pod \"484a937a-e12d-43a7-92c4-a2a42b4538d9\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") "
Sep 14 11:18:22.106911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.106745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484a937a-e12d-43a7-92c4-a2a42b4538d9-tls-certs\") pod \"484a937a-e12d-43a7-92c4-a2a42b4538d9\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") "
Sep 14 11:18:22.106911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.106771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-dshm\") pod \"484a937a-e12d-43a7-92c4-a2a42b4538d9\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") "
Sep 14 11:18:22.106911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.106809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-tmp-dir\") pod \"484a937a-e12d-43a7-92c4-a2a42b4538d9\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") "
Sep 14 11:18:22.106911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.106850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-home\") pod \"484a937a-e12d-43a7-92c4-a2a42b4538d9\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") "
Sep 14 11:18:22.107075 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.106968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-model-cache\") pod \"484a937a-e12d-43a7-92c4-a2a42b4538d9\" (UID: \"484a937a-e12d-43a7-92c4-a2a42b4538d9\") "
Sep 14 11:18:22.107176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.107145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-home" (OuterVolumeSpecName: "home") pod "484a937a-e12d-43a7-92c4-a2a42b4538d9" (UID: "484a937a-e12d-43a7-92c4-a2a42b4538d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:22.107249 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.107153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "484a937a-e12d-43a7-92c4-a2a42b4538d9" (UID: "484a937a-e12d-43a7-92c4-a2a42b4538d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:22.107306 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.107269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "484a937a-e12d-43a7-92c4-a2a42b4538d9" (UID: "484a937a-e12d-43a7-92c4-a2a42b4538d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:22.107306 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.107298    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:22.107378 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.107314    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:22.109133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.109115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484a937a-e12d-43a7-92c4-a2a42b4538d9-kube-api-access-65g6k" (OuterVolumeSpecName: "kube-api-access-65g6k") pod "484a937a-e12d-43a7-92c4-a2a42b4538d9" (UID: "484a937a-e12d-43a7-92c4-a2a42b4538d9"). InnerVolumeSpecName "kube-api-access-65g6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:22.109378 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.109354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-dshm" (OuterVolumeSpecName: "dshm") pod "484a937a-e12d-43a7-92c4-a2a42b4538d9" (UID: "484a937a-e12d-43a7-92c4-a2a42b4538d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:18:22.109378 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.109365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484a937a-e12d-43a7-92c4-a2a42b4538d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "484a937a-e12d-43a7-92c4-a2a42b4538d9" (UID: "484a937a-e12d-43a7-92c4-a2a42b4538d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:22.208686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.208655    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:22.208686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.208680    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g6k\" (UniqueName: \"kubernetes.io/projected/484a937a-e12d-43a7-92c4-a2a42b4538d9-kube-api-access-65g6k\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:22.208686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.208690    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484a937a-e12d-43a7-92c4-a2a42b4538d9-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:22.208921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.208698    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484a937a-e12d-43a7-92c4-a2a42b4538d9-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:22.624337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.624306    2563 generic.go:356] "Generic (PLEG): container finished" podID="484a937a-e12d-43a7-92c4-a2a42b4538d9" containerID="70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee" exitCode=0
Sep 14 11:18:22.624538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.624347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"
Sep 14 11:18:22.624538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.624417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b" event={"ID":"484a937a-e12d-43a7-92c4-a2a42b4538d9","Type":"ContainerDied","Data":"70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee"}
Sep 14 11:18:22.624538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.624468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b" event={"ID":"484a937a-e12d-43a7-92c4-a2a42b4538d9","Type":"ContainerDied","Data":"9a47eca637f75d7c7bc8091c7c690aacd6ddd6fbee4bb278f87fb271093962e1"}
Sep 14 11:18:22.624538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.624490    2563 scope.go:117] "RemoveContainer" containerID="70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee"
Sep 14 11:18:22.640141 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.640120    2563 scope.go:117] "RemoveContainer" containerID="70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee"
Sep 14 11:18:22.640434 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:18:22.640406    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee\": container with ID starting with 70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee not found: ID does not exist" containerID="70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee"
Sep 14 11:18:22.640541 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.640440    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee"} err="failed to get container status \"70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee\": rpc error: code = NotFound desc = could not find container \"70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee\": container with ID starting with 70047c13568212129a28a0014e8071b6a337ab6294365a54c19a65875a5575ee not found: ID does not exist"
Sep 14 11:18:22.655685 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.655636    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"]
Sep 14 11:18:22.658140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:22.658118    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-27a1a2/autores-model-27a1a2-kserve-7887b85887-dmm8b"]
Sep 14 11:18:23.919581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:23.919542    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484a937a-e12d-43a7-92c4-a2a42b4538d9" path="/var/lib/kubelet/pods/484a937a-e12d-43a7-92c4-a2a42b4538d9/volumes"
Sep 14 11:18:25.752424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:25.752376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:26.815777 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:26.815747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:28.717520 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:28.717481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:28.817966 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:28.817925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:29.158323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:29.158284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:18:29.609788 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:29.609749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:30.554047 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.553960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:30.873585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.873549    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"]
Sep 14 11:18:30.874015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.874002    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484a937a-e12d-43a7-92c4-a2a42b4538d9" containerName="main"
Sep 14 11:18:30.874062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.874017    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="484a937a-e12d-43a7-92c4-a2a42b4538d9" containerName="main"
Sep 14 11:18:30.874098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.874082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="484a937a-e12d-43a7-92c4-a2a42b4538d9" containerName="main"
Sep 14 11:18:30.878298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.878276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:30.880903 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.880878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"premium-w4-de0f7a-kserve-self-signed-certs\""
Sep 14 11:18:30.889532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.889499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"]
Sep 14 11:18:30.895572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.895539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:30.988080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.988037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-model-cache\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:30.988267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.988171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vsj\" (UniqueName: \"kubernetes.io/projected/8e086787-53a2-4d1b-b0fa-e0d167cc7138-kube-api-access-r9vsj\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:30.988267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.988220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tls-certs\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:30.988456 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.988434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-dshm\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:30.988505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.988486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tmp-dir\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:30.988560 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:30.988543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-home\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.019298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.019263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:31.090029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.089983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tmp-dir\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-home\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-model-cache\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vsj\" (UniqueName: \"kubernetes.io/projected/8e086787-53a2-4d1b-b0fa-e0d167cc7138-kube-api-access-r9vsj\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tls-certs\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-dshm\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tmp-dir\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-model-cache\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.090621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.090567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-home\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.092525 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.092501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-dshm\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.092806 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.092787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tls-certs\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.098060 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.098033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vsj\" (UniqueName: \"kubernetes.io/projected/8e086787-53a2-4d1b-b0fa-e0d167cc7138-kube-api-access-r9vsj\") pod \"premium-w4-de0f7a-kserve-564fbf8647-gcgpg\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") " pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.118454 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.118406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:31.189499 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.189385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:31.396321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.396279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:31.474811 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.474782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"]
Sep 14 11:18:31.477154 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:18:31.477122    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e086787_53a2_4d1b_b0fa_e0d167cc7138.slice/crio-9cffd8faf0a72626790c55e4523f7b9144f83950be3ec9c727079d5af5218e6f WatchSource:0}: Error finding container 9cffd8faf0a72626790c55e4523f7b9144f83950be3ec9c727079d5af5218e6f: Status 404 returned error can't find the container with id 9cffd8faf0a72626790c55e4523f7b9144f83950be3ec9c727079d5af5218e6f
Sep 14 11:18:31.679686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.679646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg" event={"ID":"8e086787-53a2-4d1b-b0fa-e0d167cc7138","Type":"ContainerStarted","Data":"b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e"}
Sep 14 11:18:31.680086 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.679693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg" event={"ID":"8e086787-53a2-4d1b-b0fa-e0d167cc7138","Type":"ContainerStarted","Data":"9cffd8faf0a72626790c55e4523f7b9144f83950be3ec9c727079d5af5218e6f"}
Sep 14 11:18:31.702068 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:31.702011    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg" podStartSLOduration=1.7019923380000002 podStartE2EDuration="1.701992338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:31.698429504 +0000 UTC m=+1060.320982715" watchObservedRunningTime="2026-09-14 11:18:31.701992338 +0000 UTC m=+1060.324545551"
Sep 14 11:18:32.304956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:32.304921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:33.074686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:33.074645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:34.076840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:34.076800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:34.932958 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:34.932915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:40.609521 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:40.609482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:41.190165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.190127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:41.190165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.190176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:41.202875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.202847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:41.371622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.371588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"]
Sep 14 11:18:41.371903 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.371855    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" podUID="806adac7-e76e-41be-8651-64f9fade816a" containerName="maas-api" containerID="cri-o://25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4" gracePeriod=30
Sep 14 11:18:41.406163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.406130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:41.502505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.502469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:41.593324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.593292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:41.626864 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.626836    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:41.686812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.686775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:41.747229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.747132    2563 generic.go:356] "Generic (PLEG): container finished" podID="806adac7-e76e-41be-8651-64f9fade816a" containerID="25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4" exitCode=0
Sep 14 11:18:41.747229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.747177    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"
Sep 14 11:18:41.747493 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.747227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" event={"ID":"806adac7-e76e-41be-8651-64f9fade816a","Type":"ContainerDied","Data":"25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4"}
Sep 14 11:18:41.747493 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.747276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5" event={"ID":"806adac7-e76e-41be-8651-64f9fade816a","Type":"ContainerDied","Data":"9faeb207fb85333967bfd57d5f2740f0533e6634f37e1db26b91d4effc699474"}
Sep 14 11:18:41.747493 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.747299    2563 scope.go:117] "RemoveContainer" containerID="25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4"
Sep 14 11:18:41.757518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.757493    2563 scope.go:117] "RemoveContainer" containerID="25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4"
Sep 14 11:18:41.757821 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:18:41.757799    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4\": container with ID starting with 25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4 not found: ID does not exist" containerID="25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4"
Sep 14 11:18:41.757898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.757829    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4"} err="failed to get container status \"25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4\": rpc error: code = NotFound desc = could not find container \"25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4\": container with ID starting with 25328f24a7a978df878f00655d463c99ee98ffdf38ab71e4435d1a0d92f9c2f4 not found: ID does not exist"
Sep 14 11:18:41.758855 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.758838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:18:41.791088 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.791060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls\") pod \"806adac7-e76e-41be-8651-64f9fade816a\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") "
Sep 14 11:18:41.791259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.791096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-metrics-certs\") pod \"806adac7-e76e-41be-8651-64f9fade816a\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") "
Sep 14 11:18:41.791259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.791166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mbpm\" (UniqueName: \"kubernetes.io/projected/806adac7-e76e-41be-8651-64f9fade816a-kube-api-access-6mbpm\") pod \"806adac7-e76e-41be-8651-64f9fade816a\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") "
Sep 14 11:18:41.791259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.791238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806adac7-e76e-41be-8651-64f9fade816a-openshift-service-ca\") pod \"806adac7-e76e-41be-8651-64f9fade816a\" (UID: \"806adac7-e76e-41be-8651-64f9fade816a\") "
Sep 14 11:18:41.791750 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.791712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806adac7-e76e-41be-8651-64f9fade816a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "806adac7-e76e-41be-8651-64f9fade816a" (UID: "806adac7-e76e-41be-8651-64f9fade816a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:18:41.793464 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.793436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806adac7-e76e-41be-8651-64f9fade816a-kube-api-access-6mbpm" (OuterVolumeSpecName: "kube-api-access-6mbpm") pod "806adac7-e76e-41be-8651-64f9fade816a" (UID: "806adac7-e76e-41be-8651-64f9fade816a"). InnerVolumeSpecName "kube-api-access-6mbpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:18:41.793605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.793584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "806adac7-e76e-41be-8651-64f9fade816a" (UID: "806adac7-e76e-41be-8651-64f9fade816a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:41.793705 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.793689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "806adac7-e76e-41be-8651-64f9fade816a" (UID: "806adac7-e76e-41be-8651-64f9fade816a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:18:41.892274 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.892220    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:41.892274 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.892269    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806adac7-e76e-41be-8651-64f9fade816a-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:41.892540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.892284    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mbpm\" (UniqueName: \"kubernetes.io/projected/806adac7-e76e-41be-8651-64f9fade816a-kube-api-access-6mbpm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:41.892540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:41.892299    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/806adac7-e76e-41be-8651-64f9fade816a-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:18:42.065360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:42.065275    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"]
Sep 14 11:18:42.072675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:42.072647    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88bdd8bf-74855bcfcc-jjvh5"]
Sep 14 11:18:42.113798 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:42.113764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:42.866605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:42.866571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:43.118642 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:43.118537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:43.313343 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:43.313308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:43.663807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:43.663770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:43.924467 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:43.920924    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="806adac7-e76e-41be-8651-64f9fade816a" path="/var/lib/kubelet/pods/806adac7-e76e-41be-8651-64f9fade816a/volumes"
Sep 14 11:18:43.976711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:43.976671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:44.193100 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:44.192978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:45.024176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:45.024137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:45.621342 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:45.621300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:45.841794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:45.841763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:46.128322 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.128283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:46.290168 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.290123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:46.783951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.783911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:18:46.813691 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.813657    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"]
Sep 14 11:18:46.814172 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.814154    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806adac7-e76e-41be-8651-64f9fade816a" containerName="maas-api"
Sep 14 11:18:46.814172 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.814174    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="806adac7-e76e-41be-8651-64f9fade816a" containerName="maas-api"
Sep 14 11:18:46.814336 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.814282    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="806adac7-e76e-41be-8651-64f9fade816a" containerName="maas-api"
Sep 14 11:18:46.818869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.818846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.823051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.823032    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 11:18:46.824363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.824348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-41158e-openshift-default-dockercfg-897xd\""
Sep 14 11:18:46.855619 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.855588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"]
Sep 14 11:18:46.940254 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-credential-socket\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940254 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istiod-ca-cert\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-data\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-token\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-certs\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-envoy\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940498 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-podinfo\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-socket\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:46.940684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:46.940618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9tb\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-kube-api-access-tf9tb\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.041783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.041694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-certs\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.041783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.041746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-envoy\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.041783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.041781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-podinfo\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.041892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-socket\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.041936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9tb\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-kube-api-access-tf9tb\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.041986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-credential-socket\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istiod-ca-cert\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-data\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-token\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-certs\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-socket\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-data\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.042547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-credential-socket\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.043022 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.042992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istiod-ca-cert\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.044520 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.044500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-envoy\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.044637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.044619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-podinfo\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.052787 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.052766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-token\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.054574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.054549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9tb\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-kube-api-access-tf9tb\") pod \"e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") " pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.130287 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.130247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:47.134805 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.134423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:47.462920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.462894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"]
Sep 14 11:18:47.464653 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:18:47.464626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ea7079_0d97_441b_8fbe_a563cb8c8a0d.slice/crio-20267cc906b94bc38f8c44da5515191f4262b211cae68de29c77ff3fefc19e29 WatchSource:0}: Error finding container 20267cc906b94bc38f8c44da5515191f4262b211cae68de29c77ff3fefc19e29: Status 404 returned error can't find the container with id 20267cc906b94bc38f8c44da5515191f4262b211cae68de29c77ff3fefc19e29
Sep 14 11:18:47.785295 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.785205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" event={"ID":"08ea7079-0d97-441b-8fbe-a563cb8c8a0d","Type":"ContainerStarted","Data":"20267cc906b94bc38f8c44da5515191f4262b211cae68de29c77ff3fefc19e29"}
Sep 14 11:18:47.813548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.813515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:47.916707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:47.916667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:48.666700 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:48.666654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:49.485484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:49.485436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:50.409782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:50.409743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:50.821588 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:50.821550    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:18:50.821693 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:50.821627    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:18:50.821693 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:50.821665    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:18:51.435937 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:51.435903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:51.537447 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:51.537406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:51.817845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:51.817755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" event={"ID":"08ea7079-0d97-441b-8fbe-a563cb8c8a0d","Type":"ContainerStarted","Data":"bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"}
Sep 14 11:18:51.847109 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:51.847057    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podStartSLOduration=2.493004993 podStartE2EDuration="5.847041412s" totalImagesPullingTime="3.354036419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:51.843973488 +0000 UTC m=+1080.466526699" watchObservedRunningTime="2026-09-14 11:18:51.847041412 +0000 UTC m=+1080.469594626"
Sep 14 11:18:52.130607 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:52.130573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:52.136051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:52.136022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:52.240913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:52.240876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:52.359078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:52.359040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:52.824429 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:52.824399    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:52.825587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:52.825563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:18:54.050811 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:54.050778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:54.311007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:54.310928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:54.408780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:54.408739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:55.215655 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:55.215620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:56.400838 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:56.400803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:56.917171 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:56.917128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:57.515134 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:57.515094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:58.348721 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:58.348688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:18:59.332996 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:18:59.332963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:00.253946 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:00.253909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:00.355937 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:00.355903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:00.715343 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:00.715300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:00.915863 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:00.915820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:01.018783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.018687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:01.081570 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.081529    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"]
Sep 14 11:19:01.087004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.086973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:01.092445 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.092415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:19:01.092603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.092480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\""
Sep 14 11:19:01.102358 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.102320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"]
Sep 14 11:19:01.178810 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.178777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2\" (UID: \"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:01.279713 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.279621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2\" (UID: \"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:01.280303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.280281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2\" (UID: \"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:01.398062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.398020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:01.621993 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.621956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:01.677690 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.677646    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"]
Sep 14 11:19:01.678618 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:19:01.678590    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41aba5d_c08a_4ecc_90ae_c61b24ced5a9.slice/crio-28ae116d79bdab5ec077e1423aab4a7b838ac802582eeeb164eb0a4d1b662993 WatchSource:0}: Error finding container 28ae116d79bdab5ec077e1423aab4a7b838ac802582eeeb164eb0a4d1b662993: Status 404 returned error can't find the container with id 28ae116d79bdab5ec077e1423aab4a7b838ac802582eeeb164eb0a4d1b662993
Sep 14 11:19:01.878786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:01.878688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2" event={"ID":"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9","Type":"ContainerStarted","Data":"28ae116d79bdab5ec077e1423aab4a7b838ac802582eeeb164eb0a4d1b662993"}
Sep 14 11:19:02.646997 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:02.646941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:03.515470 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:03.515411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:04.514274 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:04.514185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:04.900673 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:04.900629    2563 generic.go:356] "Generic (PLEG): container finished" podID="e41aba5d-c08a-4ecc-90ae-c61b24ced5a9" containerID="2c9a3ba756767ba228cccfa34f93a9133589c1681aa495121692820af8d5206c" exitCode=0
Sep 14 11:19:04.901021 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:04.900688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2" event={"ID":"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9","Type":"ContainerDied","Data":"2c9a3ba756767ba228cccfa34f93a9133589c1681aa495121692820af8d5206c"}
Sep 14 11:19:05.436571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:05.436526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:06.045327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.045303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:06.130730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.130694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9-maas-api-service-ca\") pod \"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9\" (UID: \"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9\") "
Sep 14 11:19:06.131066 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.131042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e41aba5d-c08a-4ecc-90ae-c61b24ced5a9" (UID: "e41aba5d-c08a-4ecc-90ae-c61b24ced5a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:19:06.232460 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.232324    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9-maas-api-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:19:06.343091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.343055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:06.913931 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.913894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2" event={"ID":"e41aba5d-c08a-4ecc-90ae-c61b24ced5a9","Type":"ContainerDied","Data":"28ae116d79bdab5ec077e1423aab4a7b838ac802582eeeb164eb0a4d1b662993"}
Sep 14 11:19:06.913931 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.913914    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"
Sep 14 11:19:06.913931 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.913938    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ae116d79bdab5ec077e1423aab4a7b838ac802582eeeb164eb0a4d1b662993"
Sep 14 11:19:06.918675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:06.918636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:07.324435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:07.324315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:07.573566 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:07.573519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:19:11.227739 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:11.227703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:11.934561 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:11.934522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:12.034729 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.034684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:12.106372 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.106334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:12.202048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.201862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:12.209868 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.209841    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"]
Sep 14 11:19:12.213354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.213324    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6fcec1be-qzcf2"]
Sep 14 11:19:12.274740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.274702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:12.367828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.367787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:12.685821 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:12.685779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:13.104329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:13.104291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:13.203282 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:13.203246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:13.299759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:13.299722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:13.831298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:13.831264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:13.920104 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:13.920070    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41aba5d-c08a-4ecc-90ae-c61b24ced5a9" path="/var/lib/kubelet/pods/e41aba5d-c08a-4ecc-90ae-c61b24ced5a9/volumes"
Sep 14 11:19:14.799257 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:14.799222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:14.903662 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:14.903621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:15.602636 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:15.602600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:16.716374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:16.716336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:17.622626 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:17.622585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:19.018022 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.017929    2563 generic.go:356] "Generic (PLEG): container finished" podID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerID="bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc" exitCode=0
Sep 14 11:19:19.018022 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.017998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" event={"ID":"08ea7079-0d97-441b-8fbe-a563cb8c8a0d","Type":"ContainerDied","Data":"bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"}
Sep 14 11:19:19.018471 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.018455    2563 scope.go:117] "RemoveContainer" containerID="bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"
Sep 14 11:19:19.019644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.019616    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:19:19.019697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.019687    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:19:19.019732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.019718    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:19:19.162004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.161968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:19.312872 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:19.312794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:20.026406 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:20.026358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" event={"ID":"08ea7079-0d97-441b-8fbe-a563cb8c8a0d","Type":"ContainerStarted","Data":"921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7"}
Sep 14 11:19:20.130558 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:20.130521    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:19:20.131890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:20.131861    2563 patch_prober.go:28] interesting pod/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 14 11:19:20.132005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:20.131923    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 14 11:19:20.461943 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:20.461903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:21.131647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:21.131608    2563 patch_prober.go:28] interesting pod/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 14 11:19:21.132077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:21.131662    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 14 11:19:21.391896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:21.391801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:21.901847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:21.901807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:21.973436 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:21.973384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:22.069986 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.069952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:22.131149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.131111    2563 patch_prober.go:28] interesting pod/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 14 11:19:22.131324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.131167    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 14 11:19:22.166541 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.166452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:22.268940 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.268902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:22.371166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.371125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:22.734183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.734149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:22.824664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:22.824632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:19:23.131728 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:23.131685    2563 patch_prober.go:28] interesting pod/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 14 11:19:23.131891 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:23.131746    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 14 11:19:23.716972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:23.716931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:24.131173 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:24.131133    2563 patch_prober.go:28] interesting pod/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 14 11:19:24.131357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:24.131206    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 14 11:19:24.674844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:24.674809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:25.135140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:25.135108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:19:25.136074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:25.136056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:19:29.362350 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:29.362312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:19:32.090766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:32.090728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:48.586659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:48.586624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:48.665039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:48.665010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:19:55.575650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:55.575612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:19:57.003347 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:19:57.003316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:01.779160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.779126    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"]
Sep 14 11:20:01.779833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.779812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41aba5d-c08a-4ecc-90ae-c61b24ced5a9" containerName="revoke-keys"
Sep 14 11:20:01.779929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.779836    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41aba5d-c08a-4ecc-90ae-c61b24ced5a9" containerName="revoke-keys"
Sep 14 11:20:01.779987 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.779929    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41aba5d-c08a-4ecc-90ae-c61b24ced5a9" containerName="revoke-keys"
Sep 14 11:20:01.783365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.783342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.788264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.788217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-41158e\"/\"default-dockercfg-wwzbv\""
Sep 14 11:20:01.788264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.788242    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-41158e\"/\"explicit-model-41158e-kserve-self-signed-certs\""
Sep 14 11:20:01.788548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.788263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-41158e\"/\"kube-root-ca.crt\""
Sep 14 11:20:01.788548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.788217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-41158e\"/\"openshift-service-ca.crt\""
Sep 14 11:20:01.800440 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.800410    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"]
Sep 14 11:20:01.829614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.829574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:01.859761 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.859724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-dshm\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.859956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.859796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-home\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.859956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.859816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd094445-4b5c-4291-81f5-79870c80181d-tls-certs\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.859956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.859860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lg6f\" (UniqueName: \"kubernetes.io/projected/fd094445-4b5c-4291-81f5-79870c80181d-kube-api-access-7lg6f\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.859956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.859917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-model-cache\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.860117 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.859960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-tmp-dir\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-model-cache\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-tmp-dir\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-dshm\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-home\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd094445-4b5c-4291-81f5-79870c80181d-tls-certs\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961615 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lg6f\" (UniqueName: \"kubernetes.io/projected/fd094445-4b5c-4291-81f5-79870c80181d-kube-api-access-7lg6f\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961615 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-model-cache\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961725 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-home\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.961820 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.961800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-tmp-dir\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.963823 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.963796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-dshm\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.964132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.964111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd094445-4b5c-4291-81f5-79870c80181d-tls-certs\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:01.985722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:01.985693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lg6f\" (UniqueName: \"kubernetes.io/projected/fd094445-4b5c-4291-81f5-79870c80181d-kube-api-access-7lg6f\") pod \"explicit-model-41158e-kserve-7977cdc74f-7qn4k\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") " pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:02.093301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:02.093204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:02.381226 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:02.381202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"]
Sep 14 11:20:02.383751 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:20:02.383720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd094445_4b5c_4291_81f5_79870c80181d.slice/crio-5abd46ae937125d5e3f9cc04c682d01bff8214deb8d2b51d0a53da3fb58f518f WatchSource:0}: Error finding container 5abd46ae937125d5e3f9cc04c682d01bff8214deb8d2b51d0a53da3fb58f518f: Status 404 returned error can't find the container with id 5abd46ae937125d5e3f9cc04c682d01bff8214deb8d2b51d0a53da3fb58f518f
Sep 14 11:20:03.286034 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:03.285997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k" event={"ID":"fd094445-4b5c-4291-81f5-79870c80181d","Type":"ContainerStarted","Data":"45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f"}
Sep 14 11:20:03.286512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:03.286043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k" event={"ID":"fd094445-4b5c-4291-81f5-79870c80181d","Type":"ContainerStarted","Data":"5abd46ae937125d5e3f9cc04c682d01bff8214deb8d2b51d0a53da3fb58f518f"}
Sep 14 11:20:05.434298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:05.434256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:10.255002 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:10.254950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k" podStartSLOduration=9.254932422 podStartE2EDuration="9.254932422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:03.325889703 +0000 UTC m=+1151.948442918" watchObservedRunningTime="2026-09-14 11:20:10.254932422 +0000 UTC m=+1158.877485648"
Sep 14 11:20:10.255430 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:10.255233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:20:11.677108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:11.677073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:12.094252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:12.094158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:12.094252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:12.094218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:12.107732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:12.107702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:12.353318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:12.353290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:20:17.852583 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:17.852490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:17.911344 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:17.911311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:17.982112 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:17.982060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:18.083301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:18.083268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:28.604646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:28.604604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:29.365718 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:29.365688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:35.347151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:35.347109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:35.408284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:35.408246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:35.500206 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:35.500174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:35.598939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:35.598835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:35.693829 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:35.693785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:36.274471 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:36.274430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:36.372195 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:36.372151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:36.472354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:36.472313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:37.165837 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.165804    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"]
Sep 14 11:20:37.169419 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.169381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:37.171783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.171753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\""
Sep 14 11:20:37.171913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.171806    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:20:37.178476 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.177746    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"]
Sep 14 11:20:37.284112 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.284080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f034dc-a8b4-46c6-9691-23064c3a6021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-41158e-fm8lq\" (UID: \"d8f034dc-a8b4-46c6-9691-23064c3a6021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:37.384794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.384756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f034dc-a8b4-46c6-9691-23064c3a6021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-41158e-fm8lq\" (UID: \"d8f034dc-a8b4-46c6-9691-23064c3a6021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:37.385374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.385355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f034dc-a8b4-46c6-9691-23064c3a6021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-41158e-fm8lq\" (UID: \"d8f034dc-a8b4-46c6-9691-23064c3a6021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:37.482664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.482573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:37.758292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.758264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"]
Sep 14 11:20:37.760052 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:20:37.760026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f034dc_a8b4_46c6_9691_23064c3a6021.slice/crio-65a56c8fcd8b3120816fbb0cc844ceaa358ff42503f75ad6d1691603ef2689ff WatchSource:0}: Error finding container 65a56c8fcd8b3120816fbb0cc844ceaa358ff42503f75ad6d1691603ef2689ff: Status 404 returned error can't find the container with id 65a56c8fcd8b3120816fbb0cc844ceaa358ff42503f75ad6d1691603ef2689ff
Sep 14 11:20:37.915683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:37.915654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:38.116288 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:38.116248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:38.500782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:38.500743    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8f034dc-a8b4-46c6-9691-23064c3a6021" containerID="1e2fb9b7c4371297afae13ea75fdf96edfbf60a6f46274fe96f6ade8ffc7f3de" exitCode=0
Sep 14 11:20:38.501136 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:38.500826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq" event={"ID":"d8f034dc-a8b4-46c6-9691-23064c3a6021","Type":"ContainerDied","Data":"1e2fb9b7c4371297afae13ea75fdf96edfbf60a6f46274fe96f6ade8ffc7f3de"}
Sep 14 11:20:38.501136 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:38.500882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq" event={"ID":"d8f034dc-a8b4-46c6-9691-23064c3a6021","Type":"ContainerStarted","Data":"65a56c8fcd8b3120816fbb0cc844ceaa358ff42503f75ad6d1691603ef2689ff"}
Sep 14 11:20:38.836407 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:38.836296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:39.640017 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:39.639992    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:39.707066 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:39.707034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f034dc-a8b4-46c6-9691-23064c3a6021-maas-api-service-ca\") pod \"d8f034dc-a8b4-46c6-9691-23064c3a6021\" (UID: \"d8f034dc-a8b4-46c6-9691-23064c3a6021\") "
Sep 14 11:20:39.707381 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:39.707358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f034dc-a8b4-46c6-9691-23064c3a6021-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8f034dc-a8b4-46c6-9691-23064c3a6021" (UID: "d8f034dc-a8b4-46c6-9691-23064c3a6021"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:20:39.785283 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:39.785196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:39.808658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:39.808626    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f034dc-a8b4-46c6-9691-23064c3a6021-maas-api-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:20:40.512029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:40.511997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:40.514315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:40.514291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"
Sep 14 11:20:40.514457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:40.514317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq" event={"ID":"d8f034dc-a8b4-46c6-9691-23064c3a6021","Type":"ContainerDied","Data":"65a56c8fcd8b3120816fbb0cc844ceaa358ff42503f75ad6d1691603ef2689ff"}
Sep 14 11:20:40.514457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:40.514382    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a56c8fcd8b3120816fbb0cc844ceaa358ff42503f75ad6d1691603ef2689ff"
Sep 14 11:20:41.431630 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:41.431593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:42.201219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:42.201187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:43.040545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:43.040508    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:20:43.769608 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:43.769566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:43.870581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:43.870548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:44.424992 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:44.424952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:47.683918 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.683883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:47.765838 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.765799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:47.824587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.824552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:47.907784 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.907751    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"]
Sep 14 11:20:47.910137 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.910115    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-41158e-fm8lq"]
Sep 14 11:20:47.914707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.914676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:47.919934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.919908    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f034dc-a8b4-46c6-9691-23064c3a6021" path="/var/lib/kubelet/pods/d8f034dc-a8b4-46c6-9691-23064c3a6021/volumes"
Sep 14 11:20:47.973618 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:47.973534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:48.274785 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:48.274689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:48.374717 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:48.374685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:49.249027 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:49.248993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:49.990098 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:49.990061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:50.141526 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:50.141493    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"]
Sep 14 11:20:50.147630 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:50.147596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:50.792653 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:50.792613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:51.481766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:51.481733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:51.582280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:51.582249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:51.682571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:51.682535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:51.789280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:51.789189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:51.932083 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:51.932054    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:20:51.941941 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:51.941913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:20:52.057634 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:52.057513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:52.217682 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:52.217637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:53.351209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:53.351175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:57.233150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:57.233107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:57.991460 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:57.991421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:58.079044 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:58.079012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:58.377697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:58.377653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:20:59.127781 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:20:59.127745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:00.129839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:00.129809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:00.887055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:00.887020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:01.693522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:01.693482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:01.752414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:01.752364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:01.864209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:01.864166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:02.166085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:02.166032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:02.264370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:02.264328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:04.808039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:04.808001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:05.161366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.161310    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k" podUID="fd094445-4b5c-4291-81f5-79870c80181d" containerName="main" containerID="cri-o://45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f" gracePeriod=45
Sep 14 11:21:05.420511 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.420444    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:21:05.462957 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.462917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-dshm\") pod \"fd094445-4b5c-4291-81f5-79870c80181d\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") "
Sep 14 11:21:05.463129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.462980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-home\") pod \"fd094445-4b5c-4291-81f5-79870c80181d\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") "
Sep 14 11:21:05.463129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-model-cache\") pod \"fd094445-4b5c-4291-81f5-79870c80181d\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") "
Sep 14 11:21:05.463129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-tmp-dir\") pod \"fd094445-4b5c-4291-81f5-79870c80181d\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") "
Sep 14 11:21:05.463272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lg6f\" (UniqueName: \"kubernetes.io/projected/fd094445-4b5c-4291-81f5-79870c80181d-kube-api-access-7lg6f\") pod \"fd094445-4b5c-4291-81f5-79870c80181d\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") "
Sep 14 11:21:05.463272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd094445-4b5c-4291-81f5-79870c80181d-tls-certs\") pod \"fd094445-4b5c-4291-81f5-79870c80181d\" (UID: \"fd094445-4b5c-4291-81f5-79870c80181d\") "
Sep 14 11:21:05.463377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-home" (OuterVolumeSpecName: "home") pod "fd094445-4b5c-4291-81f5-79870c80181d" (UID: "fd094445-4b5c-4291-81f5-79870c80181d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:05.463459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd094445-4b5c-4291-81f5-79870c80181d" (UID: "fd094445-4b5c-4291-81f5-79870c80181d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:05.463459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd094445-4b5c-4291-81f5-79870c80181d" (UID: "fd094445-4b5c-4291-81f5-79870c80181d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:05.463707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463688    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:05.463707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463707    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:05.463860 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.463717    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:05.465521 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.465491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd094445-4b5c-4291-81f5-79870c80181d-kube-api-access-7lg6f" (OuterVolumeSpecName: "kube-api-access-7lg6f") pod "fd094445-4b5c-4291-81f5-79870c80181d" (UID: "fd094445-4b5c-4291-81f5-79870c80181d"). InnerVolumeSpecName "kube-api-access-7lg6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:21:05.465645 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.465622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-dshm" (OuterVolumeSpecName: "dshm") pod "fd094445-4b5c-4291-81f5-79870c80181d" (UID: "fd094445-4b5c-4291-81f5-79870c80181d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:05.465710 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.465636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd094445-4b5c-4291-81f5-79870c80181d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd094445-4b5c-4291-81f5-79870c80181d" (UID: "fd094445-4b5c-4291-81f5-79870c80181d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:21:05.564786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.564730    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lg6f\" (UniqueName: \"kubernetes.io/projected/fd094445-4b5c-4291-81f5-79870c80181d-kube-api-access-7lg6f\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:05.564786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.564780    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd094445-4b5c-4291-81f5-79870c80181d-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:05.564786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.564790    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd094445-4b5c-4291-81f5-79870c80181d-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:05.668470 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.668431    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd094445-4b5c-4291-81f5-79870c80181d" containerID="45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f" exitCode=0
Sep 14 11:21:05.668647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.668480    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"
Sep 14 11:21:05.668647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.668519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k" event={"ID":"fd094445-4b5c-4291-81f5-79870c80181d","Type":"ContainerDied","Data":"45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f"}
Sep 14 11:21:05.668647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.668567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k" event={"ID":"fd094445-4b5c-4291-81f5-79870c80181d","Type":"ContainerDied","Data":"5abd46ae937125d5e3f9cc04c682d01bff8214deb8d2b51d0a53da3fb58f518f"}
Sep 14 11:21:05.668647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.668588    2563 scope.go:117] "RemoveContainer" containerID="45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f"
Sep 14 11:21:05.681562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.681536    2563 scope.go:117] "RemoveContainer" containerID="45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f"
Sep 14 11:21:05.682035 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:21:05.682004    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f\": container with ID starting with 45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f not found: ID does not exist" containerID="45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f"
Sep 14 11:21:05.682132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.682049    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f"} err="failed to get container status \"45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f\": rpc error: code = NotFound desc = could not find container \"45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f\": container with ID starting with 45b34e7b73f758d5759453909fd491b8dd183d95245fa88d2aa6acc892ed530f not found: ID does not exist"
Sep 14 11:21:05.694998 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.694947    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"]
Sep 14 11:21:05.697910 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.697885    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-41158e/explicit-model-41158e-kserve-7977cdc74f-7qn4k"]
Sep 14 11:21:05.919707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:05.919671    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd094445-4b5c-4291-81f5-79870c80181d" path="/var/lib/kubelet/pods/fd094445-4b5c-4291-81f5-79870c80181d/volumes"
Sep 14 11:21:08.174566 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:08.174529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:13.262373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:13.262336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:14.553128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:14.553085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:21:22.635465 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:22.635421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:24.088802 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:24.088763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:24.922988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:24.922945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:25.619850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:25.619814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:26.419579 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:26.419538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:27.312745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:27.312698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:28.058666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:28.058626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:28.212688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:28.212656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:28.650174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:28.650136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:28.748572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:28.748538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:29.131585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:29.131552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:30.588851 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:30.588709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:30.821251 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:30.821213    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"]
Sep 14 11:21:30.821488 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:30.821462    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy" containerID="cri-o://921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7" gracePeriod=30
Sep 14 11:21:30.832533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:30.832498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:31.269798 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:31.269764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:31.578976 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:31.578878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:31.916753 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:31.916715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:32.799194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:32.799152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:33.276064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:33.276020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:33.374550 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:33.374507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:33.478375 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:33.478336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:34.794991 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:34.794952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:35.867479 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:35.867449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:35.968204 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:35.968170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:36.088907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.088837    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:21:36.235638 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.235607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"]
Sep 14 11:21:36.236042 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236029    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy"
Sep 14 11:21:36.236087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236044    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy"
Sep 14 11:21:36.236087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236054    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd094445-4b5c-4291-81f5-79870c80181d" containerName="main"
Sep 14 11:21:36.236087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236059    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd094445-4b5c-4291-81f5-79870c80181d" containerName="main"
Sep 14 11:21:36.236087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f034dc-a8b4-46c6-9691-23064c3a6021" containerName="revoke-keys"
Sep 14 11:21:36.236087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236071    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f034dc-a8b4-46c6-9691-23064c3a6021" containerName="revoke-keys"
Sep 14 11:21:36.236244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd094445-4b5c-4291-81f5-79870c80181d" containerName="main"
Sep 14 11:21:36.236244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236151    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f034dc-a8b4-46c6-9691-23064c3a6021" containerName="revoke-keys"
Sep 14 11:21:36.236244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236158    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy"
Sep 14 11:21:36.236244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.236166    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy"
Sep 14 11:21:36.239141 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.239125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:36.241465 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.241438    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:21:36.241603 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.241536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\""
Sep 14 11:21:36.248412 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.248336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"]
Sep 14 11:21:36.257890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.257862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-envoy\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.257907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-credential-socket\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.257939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-certs\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.257979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-token\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.257995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9tb\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-kube-api-access-tf9tb\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258051    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istiod-ca-cert\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-data\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-socket\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-podinfo\") pod \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\" (UID: \"08ea7079-0d97-441b-8fbe-a563cb8c8a0d\") "
Sep 14 11:21:36.258354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258297    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:36.258585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:36.258585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258503    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.258585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258524    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-credential-socket\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.258750 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:36.258750 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:36.258945 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.258908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:21:36.260422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.260382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:21:36.260693 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.260656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-kube-api-access-tf9tb" (OuterVolumeSpecName: "kube-api-access-tf9tb") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "kube-api-access-tf9tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:21:36.260797 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.260704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:21:36.260863 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.260847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08ea7079-0d97-441b-8fbe-a563cb8c8a0d" (UID: "08ea7079-0d97-441b-8fbe-a563cb8c8a0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:21:36.359960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.359921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8599545e-d6f2-4085-aa32-14b1737c30c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-508ed449-j2wx4\" (UID: \"8599545e-d6f2-4085-aa32-14b1737c30c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360026    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-token\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360040    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9tb\" (UniqueName: \"kubernetes.io/projected/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-kube-api-access-tf9tb\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360051    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istiod-ca-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360059    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-data\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360067    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-workload-socket\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360075    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-podinfo\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.360138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.360082    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ea7079-0d97-441b-8fbe-a563cb8c8a0d-istio-envoy\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:36.460621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.460568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8599545e-d6f2-4085-aa32-14b1737c30c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-508ed449-j2wx4\" (UID: \"8599545e-d6f2-4085-aa32-14b1737c30c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:36.461236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.461213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8599545e-d6f2-4085-aa32-14b1737c30c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-508ed449-j2wx4\" (UID: \"8599545e-d6f2-4085-aa32-14b1737c30c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:36.551285 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.551240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:36.607976 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.607942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:36.827519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.827492    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"]
Sep 14 11:21:36.830216 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:21:36.830192    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8599545e_d6f2_4085_aa32_14b1737c30c5.slice/crio-47ac3138b04bf8bb31b90c3405b5fe56a178198fa9ab2a70aa539a25e612fe41 WatchSource:0}: Error finding container 47ac3138b04bf8bb31b90c3405b5fe56a178198fa9ab2a70aa539a25e612fe41: Status 404 returned error can't find the container with id 47ac3138b04bf8bb31b90c3405b5fe56a178198fa9ab2a70aa539a25e612fe41
Sep 14 11:21:36.864185 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.864155    2563 generic.go:356] "Generic (PLEG): container finished" podID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerID="921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7" exitCode=0
Sep 14 11:21:36.864329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.864203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"
Sep 14 11:21:36.864329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.864236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" event={"ID":"08ea7079-0d97-441b-8fbe-a563cb8c8a0d","Type":"ContainerDied","Data":"921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7"}
Sep 14 11:21:36.864329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.864299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4" event={"ID":"08ea7079-0d97-441b-8fbe-a563cb8c8a0d","Type":"ContainerDied","Data":"20267cc906b94bc38f8c44da5515191f4262b211cae68de29c77ff3fefc19e29"}
Sep 14 11:21:36.864489 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.864332    2563 scope.go:117] "RemoveContainer" containerID="921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7"
Sep 14 11:21:36.865715 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.865694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4" event={"ID":"8599545e-d6f2-4085-aa32-14b1737c30c5","Type":"ContainerStarted","Data":"47ac3138b04bf8bb31b90c3405b5fe56a178198fa9ab2a70aa539a25e612fe41"}
Sep 14 11:21:36.883207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.883187    2563 scope.go:117] "RemoveContainer" containerID="bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"
Sep 14 11:21:36.892935 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.892912    2563 scope.go:117] "RemoveContainer" containerID="921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7"
Sep 14 11:21:36.893259 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:21:36.893238    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7\": container with ID starting with 921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7 not found: ID does not exist" containerID="921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7"
Sep 14 11:21:36.893304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.893270    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7"} err="failed to get container status \"921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7\": rpc error: code = NotFound desc = could not find container \"921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7\": container with ID starting with 921d4fc5929863b2ffa7a24555c91a036a884c80be1cd0fe232ea3b7dda086e7 not found: ID does not exist"
Sep 14 11:21:36.893304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.893289    2563 scope.go:117] "RemoveContainer" containerID="bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"
Sep 14 11:21:36.893606 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:21:36.893588    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc\": container with ID starting with bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc not found: ID does not exist" containerID="bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"
Sep 14 11:21:36.893689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.893609    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc"} err="failed to get container status \"bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc\": rpc error: code = NotFound desc = could not find container \"bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc\": container with ID starting with bc2f830ae054b14d757b3cae7552fba0543d10c40c443d09950de2bb9c85badc not found: ID does not exist"
Sep 14 11:21:36.898232 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.898206    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"]
Sep 14 11:21:36.911972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:36.911943    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-41158e-openshift-default-58c8f7f4d8-t56f4"]
Sep 14 11:21:37.147331 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.147294    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"]
Sep 14 11:21:37.147783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.147769    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy"
Sep 14 11:21:37.147830 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.147785    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" containerName="istio-proxy"
Sep 14 11:21:37.151001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.150978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.155210 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.155186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"e2e-conflict-a-921ff281-kserve-self-signed-certs\""
Sep 14 11:21:37.176151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.176113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"]
Sep 14 11:21:37.268815 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.268765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhrn\" (UniqueName: \"kubernetes.io/projected/f2ca1660-436c-46b4-b914-d5c936a987f9-kube-api-access-shhrn\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.268815 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.268826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca1660-436c-46b4-b914-d5c936a987f9-tls-certs\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.269069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.268901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-tmp-dir\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.269069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.268922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-home\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.269069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.268971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-dshm\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.269069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.269008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-model-cache\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.292282 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.292243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:37.370537 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-dshm\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.370537 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-model-cache\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.370778 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhrn\" (UniqueName: \"kubernetes.io/projected/f2ca1660-436c-46b4-b914-d5c936a987f9-kube-api-access-shhrn\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.370778 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca1660-436c-46b4-b914-d5c936a987f9-tls-certs\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.370778 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-tmp-dir\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.370778 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-home\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.371023 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.370998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-model-cache\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.371094 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.371060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-home\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.371147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.371099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-tmp-dir\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.373013 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.372988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-dshm\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.373276 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.373255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca1660-436c-46b4-b914-d5c936a987f9-tls-certs\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.383077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.383050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhrn\" (UniqueName: \"kubernetes.io/projected/f2ca1660-436c-46b4-b914-d5c936a987f9-kube-api-access-shhrn\") pod \"e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.460278 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.460243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:37.645899 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.645855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:37.740674 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.740642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"]
Sep 14 11:21:37.742214 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:21:37.742191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ca1660_436c_46b4_b914_d5c936a987f9.slice/crio-5d96e8e1b5a926571fb44380382821911478f6db1d83397bd80436a8ce69f66e WatchSource:0}: Error finding container 5d96e8e1b5a926571fb44380382821911478f6db1d83397bd80436a8ce69f66e: Status 404 returned error can't find the container with id 5d96e8e1b5a926571fb44380382821911478f6db1d83397bd80436a8ce69f66e
Sep 14 11:21:37.878007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.877974    2563 generic.go:356] "Generic (PLEG): container finished" podID="8599545e-d6f2-4085-aa32-14b1737c30c5" containerID="3a991a9a94a84f2ab0bac4bef177f12a57a9027a49425d4a3168c0ae7287a1aa" exitCode=0
Sep 14 11:21:37.878176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.878060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4" event={"ID":"8599545e-d6f2-4085-aa32-14b1737c30c5","Type":"ContainerDied","Data":"3a991a9a94a84f2ab0bac4bef177f12a57a9027a49425d4a3168c0ae7287a1aa"}
Sep 14 11:21:37.880602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.880570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6" event={"ID":"f2ca1660-436c-46b4-b914-d5c936a987f9","Type":"ContainerStarted","Data":"64e77723f98497a58aa8bb2ca83892f4b7ee4fe2ef66a5f5b5a548b2aa9ddd04"}
Sep 14 11:21:37.880734 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.880615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6" event={"ID":"f2ca1660-436c-46b4-b914-d5c936a987f9","Type":"ContainerStarted","Data":"5d96e8e1b5a926571fb44380382821911478f6db1d83397bd80436a8ce69f66e"}
Sep 14 11:21:37.918350 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.918304    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6" podStartSLOduration=0.918287982 podStartE2EDuration="918.287982ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:37.916285869 +0000 UTC m=+1246.538839082" watchObservedRunningTime="2026-09-14 11:21:37.918287982 +0000 UTC m=+1246.540841196"
Sep 14 11:21:37.920808 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:37.920782    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ea7079-0d97-441b-8fbe-a563cb8c8a0d" path="/var/lib/kubelet/pods/08ea7079-0d97-441b-8fbe-a563cb8c8a0d/volumes"
Sep 14 11:21:38.578555 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:38.578516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:39.025730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.025704    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:39.187188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.187081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8599545e-d6f2-4085-aa32-14b1737c30c5-maas-api-service-ca\") pod \"8599545e-d6f2-4085-aa32-14b1737c30c5\" (UID: \"8599545e-d6f2-4085-aa32-14b1737c30c5\") "
Sep 14 11:21:39.187503 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.187477    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8599545e-d6f2-4085-aa32-14b1737c30c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8599545e-d6f2-4085-aa32-14b1737c30c5" (UID: "8599545e-d6f2-4085-aa32-14b1737c30c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:21:39.288170 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.288128    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8599545e-d6f2-4085-aa32-14b1737c30c5-maas-api-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:21:39.401464 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.401425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:39.895320 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.895287    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"
Sep 14 11:21:39.895320 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.895313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4" event={"ID":"8599545e-d6f2-4085-aa32-14b1737c30c5","Type":"ContainerDied","Data":"47ac3138b04bf8bb31b90c3405b5fe56a178198fa9ab2a70aa539a25e612fe41"}
Sep 14 11:21:39.895559 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:39.895363    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47ac3138b04bf8bb31b90c3405b5fe56a178198fa9ab2a70aa539a25e612fe41"
Sep 14 11:21:40.203843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:40.203732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:40.908914 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:40.908877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:41.687165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:41.687128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:46.878160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:46.878066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:46.964337 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:46.964287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:47.049272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.049233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:47.170927 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.170835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:47.239149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.239109    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"]
Sep 14 11:21:47.254556 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.254509    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-508ed449-j2wx4"]
Sep 14 11:21:47.259990 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.259954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:47.461508 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.461421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:47.461508 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.461481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:47.474239 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.474204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:47.610746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.610684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:47.856270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.856237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:47.918991 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.918955    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8599545e-d6f2-4085-aa32-14b1737c30c5" path="/var/lib/kubelet/pods/8599545e-d6f2-4085-aa32-14b1737c30c5/volumes"
Sep 14 11:21:47.963840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:47.963808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:21:49.072827 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:49.072793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:49.484820 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:49.484778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:21:50.370328 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:50.370290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:51.350658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:51.350617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:52.228819 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:52.228780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:53.721519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:53.721483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:53.779548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:53.779506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:53.989578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:53.989492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:54.306806 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:54.306711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:54.404607 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:54.404568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:21:54.509892 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:21:54.509854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:00.820268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:00.820216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:00.936635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:00.936593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:01.039902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:01.039860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:01.138352 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:01.138317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:02.947209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.947169    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"]
Sep 14 11:22:02.947881 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.947858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8599545e-d6f2-4085-aa32-14b1737c30c5" containerName="revoke-keys"
Sep 14 11:22:02.947970 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.947884    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8599545e-d6f2-4085-aa32-14b1737c30c5" containerName="revoke-keys"
Sep 14 11:22:02.948024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.947992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8599545e-d6f2-4085-aa32-14b1737c30c5" containerName="revoke-keys"
Sep 14 11:22:02.952572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.952546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:02.955565 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.955539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8dd3f5\"/\"kube-root-ca.crt\""
Sep 14 11:22:02.956442 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.956422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8dd3f5\"/\"openshift-service-ca.crt\""
Sep 14 11:22:02.956515 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.956459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8dd3f5\"/\"default-dockercfg-mcshs\""
Sep 14 11:22:02.956578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.956528    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8dd3f5\"/\"noten-model-8dd3f5-kserve-self-signed-certs\""
Sep 14 11:22:02.962284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.962257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"]
Sep 14 11:22:02.976119 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:02.976079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:03.017598 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.017555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-model-cache\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.017792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.017610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-home\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.017792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.017629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-dshm\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.017792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.017685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-tmp-dir\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.017792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.017715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr4t\" (UniqueName: \"kubernetes.io/projected/1c623f37-5933-443c-bbc0-d11b91bdb61d-kube-api-access-lnr4t\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.017792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.017744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c623f37-5933-443c-bbc0-d11b91bdb61d-tls-certs\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118430 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-model-cache\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-home\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-dshm\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-tmp-dir\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr4t\" (UniqueName: \"kubernetes.io/projected/1c623f37-5933-443c-bbc0-d11b91bdb61d-kube-api-access-lnr4t\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c623f37-5933-443c-bbc0-d11b91bdb61d-tls-certs\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-model-cache\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118927 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-tmp-dir\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.118965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.118923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-home\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.120841 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.120818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-dshm\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.121193 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.121175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c623f37-5933-443c-bbc0-d11b91bdb61d-tls-certs\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.127460 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.127430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr4t\" (UniqueName: \"kubernetes.io/projected/1c623f37-5933-443c-bbc0-d11b91bdb61d-kube-api-access-lnr4t\") pod \"noten-model-8dd3f5-kserve-78f9c695f5-k6dqr\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") " pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.183710 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.183670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:03.264329 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.264194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:03.548477 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:03.548443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"]
Sep 14 11:22:03.550369 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:22:03.550337    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c623f37_5933_443c_bbc0_d11b91bdb61d.slice/crio-77f9233958858bb7d1085a7adb259e6141fcc669a4601561bf588c85efb9d820 WatchSource:0}: Error finding container 77f9233958858bb7d1085a7adb259e6141fcc669a4601561bf588c85efb9d820: Status 404 returned error can't find the container with id 77f9233958858bb7d1085a7adb259e6141fcc669a4601561bf588c85efb9d820
Sep 14 11:22:04.050795 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:04.050751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr" event={"ID":"1c623f37-5933-443c-bbc0-d11b91bdb61d","Type":"ContainerStarted","Data":"5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647"}
Sep 14 11:22:04.051207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:04.050805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr" event={"ID":"1c623f37-5933-443c-bbc0-d11b91bdb61d","Type":"ContainerStarted","Data":"77f9233958858bb7d1085a7adb259e6141fcc669a4601561bf588c85efb9d820"}
Sep 14 11:22:04.069633 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:04.069578    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr" podStartSLOduration=2.069559419 podStartE2EDuration="2.069559419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:04.066646426 +0000 UTC m=+1272.689199639" watchObservedRunningTime="2026-09-14 11:22:04.069559419 +0000 UTC m=+1272.692112664"
Sep 14 11:22:07.328657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:07.328618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:07.392175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:07.392131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:08.202676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:08.202639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:08.559538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:08.559426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:09.000532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:09.000492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:11.371884 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:11.371847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:11.996820 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:11.996779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:12.093038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:12.092995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:12.193404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:12.193369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:12.992485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:12.992453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:13.265234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:13.265126    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:13.265234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:13.265179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:13.278126 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:13.278096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:13.942664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:13.942631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:14.123890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:14.123853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:14.701858 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:14.701828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:15.706197 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:15.706159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:15.795652 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:15.795612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:16.144029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:16.143990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:16.326400 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:16.326365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:16.425745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:16.425640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:17.067001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:17.066966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:22:35.451896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:35.451851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:35.655666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:35.655624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:35.782636 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:35.782527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:38.743674 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:38.743631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:39.917533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:39.917496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:40.771907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:40.771860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"]
Sep 14 11:22:40.873345 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:40.873304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:40.959341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:40.959300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:41.983601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:41.983566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:42.089864 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:42.089826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:42.383314 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:42.383274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:42.798115 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:42.798006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:42.927363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:42.927317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:43.031883 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:43.031850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:44.164965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:44.164928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:45.370318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:45.370283    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:22:46.326722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:46.326685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:46.507200 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:46.507168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:47.634066 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:47.634028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:48.885847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:48.885810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:49.827253 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:49.827215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:50.701685 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:50.701640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:51.797379 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.797340    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"]
Sep 14 11:22:51.801622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.801596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.804926 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.804903    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:22:51.805065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.804908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-41ae25-serving-cert\""
Sep 14 11:22:51.805065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.805051    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:22:51.805200 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.804916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\""
Sep 14 11:22:51.805290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.805264    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:22:51.818879 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.818849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"]
Sep 14 11:22:51.852229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.852194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:51.895526 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.895483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkxz\" (UniqueName: \"kubernetes.io/projected/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-kube-api-access-xjkxz\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.895730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.895568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.895730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.895620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.895844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.895744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.951789 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.951754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:51.996635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.996590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.996850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.996716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.996850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.996754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkxz\" (UniqueName: \"kubernetes.io/projected/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-kube-api-access-xjkxz\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.996850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.996811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.997611 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.997589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.999256 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.999234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:51.999349 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:51.999281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:52.009684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.009647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkxz\" (UniqueName: \"kubernetes.io/projected/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-kube-api-access-xjkxz\") pod \"maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:52.051240 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.051146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:52.114921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.114879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:52.401018 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.400991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"]
Sep 14 11:22:52.402435 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:22:52.402408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b810f95_2179_462d_b8ea_0c2dbd88f5ab.slice/crio-34cb1ae7ade6d350af8cbc1b2efcd47ab870eed0ca82ad8e570c4365b6557251 WatchSource:0}: Error finding container 34cb1ae7ade6d350af8cbc1b2efcd47ab870eed0ca82ad8e570c4365b6557251: Status 404 returned error can't find the container with id 34cb1ae7ade6d350af8cbc1b2efcd47ab870eed0ca82ad8e570c4365b6557251
Sep 14 11:22:52.419635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.419596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:52.519550 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.519514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:52.618007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.617968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"]
Sep 14 11:22:52.622891 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.622864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.626807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.626781    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-41ae25\""
Sep 14 11:22:52.626926 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.626791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-41ae25-dockercfg-kxvfj\""
Sep 14 11:22:52.641826 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.641795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"]
Sep 14 11:22:52.704354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.704257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50945937-89ab-44c7-9554-2b844f715778-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.704354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.704308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxx9\" (UniqueName: \"kubernetes.io/projected/50945937-89ab-44c7-9554-2b844f715778-kube-api-access-bnxx9\") pod \"payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.805729 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.805700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxx9\" (UniqueName: \"kubernetes.io/projected/50945937-89ab-44c7-9554-2b844f715778-kube-api-access-bnxx9\") pod \"payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.806061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.805825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50945937-89ab-44c7-9554-2b844f715778-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.806484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.806465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50945937-89ab-44c7-9554-2b844f715778-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.815484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.815463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxx9\" (UniqueName: \"kubernetes.io/projected/50945937-89ab-44c7-9554-2b844f715778-kube-api-access-bnxx9\") pod \"payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:52.934186 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:52.934141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:53.227009 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.226975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"]
Sep 14 11:22:53.228739 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:22:53.228707    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50945937_89ab_44c7_9554_2b844f715778.slice/crio-f8b614c9f702e14aaa6c57a89b19ed13b4da71c87de3f180357c0b2caf32de90 WatchSource:0}: Error finding container f8b614c9f702e14aaa6c57a89b19ed13b4da71c87de3f180357c0b2caf32de90: Status 404 returned error can't find the container with id f8b614c9f702e14aaa6c57a89b19ed13b4da71c87de3f180357c0b2caf32de90
Sep 14 11:22:53.360782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.360742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6" event={"ID":"1b810f95-2179-462d-b8ea-0c2dbd88f5ab","Type":"ContainerStarted","Data":"708c78c3e8f6c7b50ba333ecf38b410f9f0d93223857ef4dcb0bb1d3fc888649"}
Sep 14 11:22:53.361004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.360812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6" event={"ID":"1b810f95-2179-462d-b8ea-0c2dbd88f5ab","Type":"ContainerStarted","Data":"34cb1ae7ade6d350af8cbc1b2efcd47ab870eed0ca82ad8e570c4365b6557251"}
Sep 14 11:22:53.361004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.360866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:22:53.362581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.362554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2" event={"ID":"50945937-89ab-44c7-9554-2b844f715778","Type":"ContainerStarted","Data":"b8bde304651ac7ef3251c266f5249d68a697f811f98c2424dabeebb478c76e4a"}
Sep 14 11:22:53.362741 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.362598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2" event={"ID":"50945937-89ab-44c7-9554-2b844f715778","Type":"ContainerStarted","Data":"f8b614c9f702e14aaa6c57a89b19ed13b4da71c87de3f180357c0b2caf32de90"}
Sep 14 11:22:53.362741 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.362670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:22:53.381480 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.381415    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6" podStartSLOduration=2.031226841 podStartE2EDuration="2.381376442s" totalImagesPullingTime="350.149601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:53.380354097 +0000 UTC m=+1322.002907320" watchObservedRunningTime="2026-09-14 11:22:53.381376442 +0000 UTC m=+1322.003929657"
Sep 14 11:22:53.397625 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:53.397555    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2" podStartSLOduration=1.397534148 podStartE2EDuration="1.397534148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:53.39710584 +0000 UTC m=+1322.019659057" watchObservedRunningTime="2026-09-14 11:22:53.397534148 +0000 UTC m=+1322.020087362"
Sep 14 11:22:54.241863 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.241822    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"]
Sep 14 11:22:54.245588 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.245568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.247929 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.247909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6f3d0f\"/\"distinct-w0-6f3d0f-kserve-self-signed-certs\""
Sep 14 11:22:54.265459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.265420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"]
Sep 14 11:22:54.321038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.320997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-model-cache\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.321216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.321047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k5l\" (UniqueName: \"kubernetes.io/projected/b307312c-c91a-4039-a029-40d098e3e3a5-kube-api-access-n4k5l\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.321216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.321150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-dshm\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.321216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.321188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-home\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.321321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.321262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b307312c-c91a-4039-a029-40d098e3e3a5-tls-certs\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.321321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.321307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-tmp-dir\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.422807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.422768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-dshm\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.422807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.422811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-home\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.422915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b307312c-c91a-4039-a029-40d098e3e3a5-tls-certs\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.422947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-tmp-dir\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.422991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-model-cache\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.423018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k5l\" (UniqueName: \"kubernetes.io/projected/b307312c-c91a-4039-a029-40d098e3e3a5-kube-api-access-n4k5l\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.423370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-home\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.423403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-model-cache\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.423621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.423442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-tmp-dir\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.425202 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.425177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-dshm\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.425413 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.425380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b307312c-c91a-4039-a029-40d098e3e3a5-tls-certs\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.431832 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.431809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k5l\" (UniqueName: \"kubernetes.io/projected/b307312c-c91a-4039-a029-40d098e3e3a5-kube-api-access-n4k5l\") pod \"distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") " pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.555580 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.555495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:22:54.662858 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.662823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:22:54.855038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:54.855012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"]
Sep 14 11:22:54.857077 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:22:54.857038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb307312c_c91a_4039_a029_40d098e3e3a5.slice/crio-d6eae5cc6d141e34289f0ff668096c4b3d38d42792f28f49469fe38c3cbeb85e WatchSource:0}: Error finding container d6eae5cc6d141e34289f0ff668096c4b3d38d42792f28f49469fe38c3cbeb85e: Status 404 returned error can't find the container with id d6eae5cc6d141e34289f0ff668096c4b3d38d42792f28f49469fe38c3cbeb85e
Sep 14 11:22:55.377365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:55.377327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx" event={"ID":"b307312c-c91a-4039-a029-40d098e3e3a5","Type":"ContainerStarted","Data":"4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1"}
Sep 14 11:22:55.377758 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:55.377376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx" event={"ID":"b307312c-c91a-4039-a029-40d098e3e3a5","Type":"ContainerStarted","Data":"d6eae5cc6d141e34289f0ff668096c4b3d38d42792f28f49469fe38c3cbeb85e"}
Sep 14 11:22:55.397244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:55.397180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx" podStartSLOduration=1.3971589739999999 podStartE2EDuration="1.397158974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:55.395769578 +0000 UTC m=+1324.018322791" watchObservedRunningTime="2026-09-14 11:22:55.397158974 +0000 UTC m=+1324.019712190"
Sep 14 11:22:55.799442 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:55.799314    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr" podUID="1c623f37-5933-443c-bbc0-d11b91bdb61d" containerName="main" containerID="cri-o://5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647" gracePeriod=45
Sep 14 11:22:56.061160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.061090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:56.138814 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.138782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-home\") pod \"1c623f37-5933-443c-bbc0-d11b91bdb61d\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") "
Sep 14 11:22:56.139014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.138841    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-model-cache\") pod \"1c623f37-5933-443c-bbc0-d11b91bdb61d\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") "
Sep 14 11:22:56.139014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.138867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnr4t\" (UniqueName: \"kubernetes.io/projected/1c623f37-5933-443c-bbc0-d11b91bdb61d-kube-api-access-lnr4t\") pod \"1c623f37-5933-443c-bbc0-d11b91bdb61d\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") "
Sep 14 11:22:56.139014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.138961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c623f37-5933-443c-bbc0-d11b91bdb61d-tls-certs\") pod \"1c623f37-5933-443c-bbc0-d11b91bdb61d\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") "
Sep 14 11:22:56.139014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.138985    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-tmp-dir\") pod \"1c623f37-5933-443c-bbc0-d11b91bdb61d\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") "
Sep 14 11:22:56.139225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.139033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-dshm\") pod \"1c623f37-5933-443c-bbc0-d11b91bdb61d\" (UID: \"1c623f37-5933-443c-bbc0-d11b91bdb61d\") "
Sep 14 11:22:56.139225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.139086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-home" (OuterVolumeSpecName: "home") pod "1c623f37-5933-443c-bbc0-d11b91bdb61d" (UID: "1c623f37-5933-443c-bbc0-d11b91bdb61d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:22:56.139225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.139168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c623f37-5933-443c-bbc0-d11b91bdb61d" (UID: "1c623f37-5933-443c-bbc0-d11b91bdb61d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:22:56.139409 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.139356    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:22:56.139497 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.139413    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:22:56.139558 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.139487    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c623f37-5933-443c-bbc0-d11b91bdb61d" (UID: "1c623f37-5933-443c-bbc0-d11b91bdb61d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:22:56.141272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.141244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c623f37-5933-443c-bbc0-d11b91bdb61d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c623f37-5933-443c-bbc0-d11b91bdb61d" (UID: "1c623f37-5933-443c-bbc0-d11b91bdb61d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:22:56.141380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.141269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-dshm" (OuterVolumeSpecName: "dshm") pod "1c623f37-5933-443c-bbc0-d11b91bdb61d" (UID: "1c623f37-5933-443c-bbc0-d11b91bdb61d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:22:56.141380 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.141340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c623f37-5933-443c-bbc0-d11b91bdb61d-kube-api-access-lnr4t" (OuterVolumeSpecName: "kube-api-access-lnr4t") pod "1c623f37-5933-443c-bbc0-d11b91bdb61d" (UID: "1c623f37-5933-443c-bbc0-d11b91bdb61d"). InnerVolumeSpecName "kube-api-access-lnr4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:22:56.240920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.240877    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c623f37-5933-443c-bbc0-d11b91bdb61d-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:22:56.240920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.240913    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:22:56.240920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.240921    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c623f37-5933-443c-bbc0-d11b91bdb61d-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:22:56.240920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.240930    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnr4t\" (UniqueName: \"kubernetes.io/projected/1c623f37-5933-443c-bbc0-d11b91bdb61d-kube-api-access-lnr4t\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:22:56.384872 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.384837    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c623f37-5933-443c-bbc0-d11b91bdb61d" containerID="5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647" exitCode=0
Sep 14 11:22:56.385341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.384882    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"
Sep 14 11:22:56.385341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.384918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr" event={"ID":"1c623f37-5933-443c-bbc0-d11b91bdb61d","Type":"ContainerDied","Data":"5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647"}
Sep 14 11:22:56.385341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.384961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr" event={"ID":"1c623f37-5933-443c-bbc0-d11b91bdb61d","Type":"ContainerDied","Data":"77f9233958858bb7d1085a7adb259e6141fcc669a4601561bf588c85efb9d820"}
Sep 14 11:22:56.385341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.384978    2563 scope.go:117] "RemoveContainer" containerID="5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647"
Sep 14 11:22:56.395235 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.395217    2563 scope.go:117] "RemoveContainer" containerID="5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647"
Sep 14 11:22:56.395560 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:22:56.395542    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647\": container with ID starting with 5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647 not found: ID does not exist" containerID="5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647"
Sep 14 11:22:56.395623 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.395567    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647"} err="failed to get container status \"5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647\": rpc error: code = NotFound desc = could not find container \"5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647\": container with ID starting with 5645782c45bee32f1fe1a8d797a388e43532bb5bbc0421bbdde8c0e64b70f647 not found: ID does not exist"
Sep 14 11:22:56.412758 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.412720    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"]
Sep 14 11:22:56.418209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:56.418176    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8dd3f5/noten-model-8dd3f5-kserve-78f9c695f5-k6dqr"]
Sep 14 11:22:57.919972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:57.919937    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c623f37-5933-443c-bbc0-d11b91bdb61d" path="/var/lib/kubelet/pods/1c623f37-5933-443c-bbc0-d11b91bdb61d/volumes"
Sep 14 11:22:59.375547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:22:59.375515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:23:02.149729 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:02.149683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:03.189169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:03.189132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:04.371604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:04.371570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:23:04.556419 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:04.556356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:23:04.556602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:04.556436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:23:04.569810 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:04.569774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:23:05.457724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:05.457693    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:23:15.598826 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:15.598737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:16.232679 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.232632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:16.376930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.376886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:16.682544 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.682505    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"]
Sep 14 11:23:16.682982 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.682969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c623f37-5933-443c-bbc0-d11b91bdb61d" containerName="main"
Sep 14 11:23:16.682982 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.682983    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c623f37-5933-443c-bbc0-d11b91bdb61d" containerName="main"
Sep 14 11:23:16.683072 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.683063    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c623f37-5933-443c-bbc0-d11b91bdb61d" containerName="main"
Sep 14 11:23:16.687607 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.687583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.690054 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.690031    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"e2e-conflict-b-921ff281-kserve-self-signed-certs\""
Sep 14 11:23:16.699080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.699054    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"]
Sep 14 11:23:16.705462 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.705431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:16.742122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.742069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-home\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.742350 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.742128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tls-certs\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.742350 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.742164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-dshm\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.742350 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.742212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfj25\" (UniqueName: \"kubernetes.io/projected/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-kube-api-access-wfj25\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.742350 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.742277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tmp-dir\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.742628 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.742480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-model-cache\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.843589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-model-cache\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.843589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-home\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.843869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tls-certs\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.843869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-dshm\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.843869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj25\" (UniqueName: \"kubernetes.io/projected/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-kube-api-access-wfj25\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.843869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tmp-dir\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.844092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.843982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-home\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.844092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.844064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-model-cache\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.844180 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.844162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tmp-dir\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.846057 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.846033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-dshm\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.846247 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.846230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tls-certs\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.852128 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.852102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj25\" (UniqueName: \"kubernetes.io/projected/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-kube-api-access-wfj25\") pod \"e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") " pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:16.998567 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:16.998470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:17.263704 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:17.263615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:23:17.284096 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:17.284056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"]
Sep 14 11:23:17.285438 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:23:17.285412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd38a6f59_ec08_424a_a065_8b6e2d76ab1c.slice/crio-6214436a139f220a154434b0d2679ee1557e680b63de0d50f32b2fd89c133801 WatchSource:0}: Error finding container 6214436a139f220a154434b0d2679ee1557e680b63de0d50f32b2fd89c133801: Status 404 returned error can't find the container with id 6214436a139f220a154434b0d2679ee1557e680b63de0d50f32b2fd89c133801
Sep 14 11:23:17.371033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:17.370981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:17.521531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:17.521425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22" event={"ID":"d38a6f59-ec08-424a-a065-8b6e2d76ab1c","Type":"ContainerStarted","Data":"6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332"}
Sep 14 11:23:17.521531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:17.521473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22" event={"ID":"d38a6f59-ec08-424a-a065-8b6e2d76ab1c","Type":"ContainerStarted","Data":"6214436a139f220a154434b0d2679ee1557e680b63de0d50f32b2fd89c133801"}
Sep 14 11:23:17.541053 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:17.540998    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22" podStartSLOduration=1.540979662 podStartE2EDuration="1.540979662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:17.539760103 +0000 UTC m=+1346.162313318" watchObservedRunningTime="2026-09-14 11:23:17.540979662 +0000 UTC m=+1346.163532876"
Sep 14 11:23:18.329290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:18.329253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:19.685881 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:19.685847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:19.763142 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:19.763093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:20.762612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:20.762572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:20.948117 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:20.948084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:22.021346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:22.021309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:23.083120 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:23.083075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:23.960836 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:23.960801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:25.060338 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:25.060294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:26.999688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:26.999644    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:26.999688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:26.999702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:27.012690 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:27.012653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:27.222550 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:27.222507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:27.595908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:27.595877    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:23:28.004731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.004689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:28.061724 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.061683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"]
Sep 14 11:23:28.061981 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.061947    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth" podUID="6a908caf-391a-4a14-b1a2-d84a0d99afa8" containerName="payload-pre-processing" containerID="cri-o://372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2" gracePeriod=30
Sep 14 11:23:28.097448 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.097404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:28.207610 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.207575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:28.302491 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.302401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:28.337064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.337034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:23:28.396321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.396284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:28.465144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.465106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a908caf-391a-4a14-b1a2-d84a0d99afa8-plugins-config-volume\") pod \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") "
Sep 14 11:23:28.465324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.465161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b59m\" (UniqueName: \"kubernetes.io/projected/6a908caf-391a-4a14-b1a2-d84a0d99afa8-kube-api-access-6b59m\") pod \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\" (UID: \"6a908caf-391a-4a14-b1a2-d84a0d99afa8\") "
Sep 14 11:23:28.465562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.465536    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a908caf-391a-4a14-b1a2-d84a0d99afa8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a908caf-391a-4a14-b1a2-d84a0d99afa8" (UID: "6a908caf-391a-4a14-b1a2-d84a0d99afa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:23:28.467371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.467344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a908caf-391a-4a14-b1a2-d84a0d99afa8-kube-api-access-6b59m" (OuterVolumeSpecName: "kube-api-access-6b59m") pod "6a908caf-391a-4a14-b1a2-d84a0d99afa8" (UID: "6a908caf-391a-4a14-b1a2-d84a0d99afa8"). InnerVolumeSpecName "kube-api-access-6b59m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:23:28.566965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.566837    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a908caf-391a-4a14-b1a2-d84a0d99afa8-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:23:28.566965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.566895    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b59m\" (UniqueName: \"kubernetes.io/projected/6a908caf-391a-4a14-b1a2-d84a0d99afa8-kube-api-access-6b59m\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:23:28.592732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.592697    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a908caf-391a-4a14-b1a2-d84a0d99afa8" containerID="372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2" exitCode=0
Sep 14 11:23:28.592920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.592744    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"
Sep 14 11:23:28.592920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.592779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth" event={"ID":"6a908caf-391a-4a14-b1a2-d84a0d99afa8","Type":"ContainerDied","Data":"372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2"}
Sep 14 11:23:28.592920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.592828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth" event={"ID":"6a908caf-391a-4a14-b1a2-d84a0d99afa8","Type":"ContainerDied","Data":"6b41fb479158c3b16df92dcd9e2902664c1eb6ff4d76b9043bbf6d76114c1c34"}
Sep 14 11:23:28.592920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.592850    2563 scope.go:117] "RemoveContainer" containerID="372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2"
Sep 14 11:23:28.602822 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.602797    2563 scope.go:117] "RemoveContainer" containerID="372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2"
Sep 14 11:23:28.603146 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:23:28.603125    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2\": container with ID starting with 372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2 not found: ID does not exist" containerID="372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2"
Sep 14 11:23:28.603190 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.603157    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2"} err="failed to get container status \"372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2\": rpc error: code = NotFound desc = could not find container \"372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2\": container with ID starting with 372fda186c96a0d0edf3cea7a18fe73ddb5d2fd4b278886e35faefb3ad745fe2 not found: ID does not exist"
Sep 14 11:23:28.654217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.654164    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"]
Sep 14 11:23:28.670998 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.670961    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6d272c-7768577cd-82rth"]
Sep 14 11:23:28.724771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.724731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:28.900577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:28.900537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:29.035501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:29.035461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:29.296491 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:29.296368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:29.572006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:29.571913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:29.919715 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:29.919676    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a908caf-391a-4a14-b1a2-d84a0d99afa8" path="/var/lib/kubelet/pods/6a908caf-391a-4a14-b1a2-d84a0d99afa8/volumes"
Sep 14 11:23:30.421193 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:30.421155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:31.325424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:31.325349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:32.315552 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:32.315514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:32.505833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:32.505796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:32.596990 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:32.596903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:33.394666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:33.394627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:34.263553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:34.263516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:35.387585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:35.387542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:35.475898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:35.475864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:35.573518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:35.573468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:35.679426 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:35.679309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:35.888930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:35.888887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:35.991775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:35.991679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:36.089312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:36.089275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:36.762620 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:36.762580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:37.622660 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:37.622618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:38.398025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:38.397990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:39.526353 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:39.526320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:40.429812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:40.429775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:41.350551 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:41.350515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:42.315237 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:42.315200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:43.058324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:43.058290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:23:43.123552 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:43.123514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:46.524775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:46.524732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:47.296037 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.296001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:47.324462 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.324352    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"]
Sep 14 11:23:47.327040 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.326998    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6" podUID="f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" containerName="payload-processing" containerID="cri-o://67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535" gracePeriod=30
Sep 14 11:23:47.414545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.414502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:47.513889 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.513849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:47.583803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.583762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:47.597512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.597481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:23:47.646593 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.646562    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-plugins-config-volume\") pod \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") "
Sep 14 11:23:47.646744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.646649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffkr2\" (UniqueName: \"kubernetes.io/projected/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-kube-api-access-ffkr2\") pod \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\" (UID: \"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5\") "
Sep 14 11:23:47.647061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.647021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" (UID: "f2ccf030-e9b0-4e42-b945-4d935c7b4bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:23:47.648749 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.648728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-kube-api-access-ffkr2" (OuterVolumeSpecName: "kube-api-access-ffkr2") pod "f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" (UID: "f2ccf030-e9b0-4e42-b945-4d935c7b4bc5"). InnerVolumeSpecName "kube-api-access-ffkr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:23:47.713065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.713031    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" containerID="67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535" exitCode=0
Sep 14 11:23:47.713236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.713078    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"
Sep 14 11:23:47.713236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.713113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6" event={"ID":"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5","Type":"ContainerDied","Data":"67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535"}
Sep 14 11:23:47.713236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.713168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6" event={"ID":"f2ccf030-e9b0-4e42-b945-4d935c7b4bc5","Type":"ContainerDied","Data":"079c7defa8948889e49ff723a50525470fd20cee840aae933af88fdd9d1e32f9"}
Sep 14 11:23:47.713236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.713191    2563 scope.go:117] "RemoveContainer" containerID="67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535"
Sep 14 11:23:47.725441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.725417    2563 scope.go:117] "RemoveContainer" containerID="67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535"
Sep 14 11:23:47.725921 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:23:47.725896    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535\": container with ID starting with 67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535 not found: ID does not exist" containerID="67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535"
Sep 14 11:23:47.726019 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.725929    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535"} err="failed to get container status \"67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535\": rpc error: code = NotFound desc = could not find container \"67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535\": container with ID starting with 67e297a0e56d7ce1721d9123313c956602e335f3eda123ed282de845aed7c535 not found: ID does not exist"
Sep 14 11:23:47.737312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.737274    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"]
Sep 14 11:23:47.742217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.742186    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39ab34-6c8f7dbb75-kswn6"]
Sep 14 11:23:47.747834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.747804    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:23:47.747834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.747836    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffkr2\" (UniqueName: \"kubernetes.io/projected/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5-kube-api-access-ffkr2\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:23:47.896867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.896828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:47.919275 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:47.919227    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" path="/var/lib/kubelet/pods/f2ccf030-e9b0-4e42-b945-4d935c7b4bc5/volumes"
Sep 14 11:23:48.018963 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:48.018926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:49.008683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:49.008647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:50.228868 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:50.228827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:51.223251 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:51.223208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:52.105557 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:52.105519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:53.110374 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:53.110335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:54.118451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:54.118411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:54.497411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:54.497301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:54.603765 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:54.603724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:54.813456 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:54.813345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:55.094473 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:55.094370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:55.397599 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:55.397564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:55.755033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:55.754944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:55.976053 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:55.976015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:56.226453 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:56.226414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:57.131944 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:57.131907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:57.423971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:57.423873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:58.159357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:58.159320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:58.811607 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:58.811564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:58.889675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:58.889635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:23:59.205373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:23:59.205334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:00.354159 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:00.354123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:01.984790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.984755    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"]
Sep 14 11:24:01.985204 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.985191    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a908caf-391a-4a14-b1a2-d84a0d99afa8" containerName="payload-pre-processing"
Sep 14 11:24:01.985252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.985205    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a908caf-391a-4a14-b1a2-d84a0d99afa8" containerName="payload-pre-processing"
Sep 14 11:24:01.985252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.985234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" containerName="payload-processing"
Sep 14 11:24:01.985252 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.985240    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" containerName="payload-processing"
Sep 14 11:24:01.985364 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.985309    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a908caf-391a-4a14-b1a2-d84a0d99afa8" containerName="payload-pre-processing"
Sep 14 11:24:01.985364 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.985319    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ccf030-e9b0-4e42-b945-4d935c7b4bc5" containerName="payload-processing"
Sep 14 11:24:01.990085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.990053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:01.992648 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:01.992624    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-55c025-serving-cert\""
Sep 14 11:24:02.002068 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.002035    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"]
Sep 14 11:24:02.020064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.020030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:02.091307 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.091266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vsv\" (UniqueName: \"kubernetes.io/projected/d072ca32-7a4a-4a11-8e2e-be109303b686-kube-api-access-24vsv\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.091307 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.091306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.091574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.091434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ca32-7a4a-4a11-8e2e-be109303b686-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.091574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.091456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.119085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.119050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:02.192660 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.192618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ca32-7a4a-4a11-8e2e-be109303b686-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.192660 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.192661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.192924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.192737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vsv\" (UniqueName: \"kubernetes.io/projected/d072ca32-7a4a-4a11-8e2e-be109303b686-kube-api-access-24vsv\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.192924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.192759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.192924 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:24:02.192838    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-serving-cert: secret "maas-api-e2e-shared-a-w3-55c025-serving-cert" not found
Sep 14 11:24:02.192924 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:24:02.192920    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls podName:d072ca32-7a4a-4a11-8e2e-be109303b686 nodeName:}" failed. No retries permitted until 2026-09-14 11:24:02.692900226 +0000 UTC m=+1391.315453422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls") pod "maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" (UID: "d072ca32-7a4a-4a11-8e2e-be109303b686") : secret "maas-api-e2e-shared-a-w3-55c025-serving-cert" not found
Sep 14 11:24:02.193433 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.193406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ca32-7a4a-4a11-8e2e-be109303b686-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.195333 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.195310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.203947 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.203911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vsv\" (UniqueName: \"kubernetes.io/projected/d072ca32-7a4a-4a11-8e2e-be109303b686-kube-api-access-24vsv\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.583411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.583353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:02.693803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.693763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:02.696953 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.696928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.699580 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.699555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:02.791767 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.791729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:02.902900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:02.902857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:03.184072 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.184045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"]
Sep 14 11:24:03.185249 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:24:03.185225    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd072ca32_7a4a_4a11_8e2e_be109303b686.slice/crio-2de199dd47cba22ceec589e722805650805367169f65299e8bf60a2e3bc632f9 WatchSource:0}: Error finding container 2de199dd47cba22ceec589e722805650805367169f65299e8bf60a2e3bc632f9: Status 404 returned error can't find the container with id 2de199dd47cba22ceec589e722805650805367169f65299e8bf60a2e3bc632f9
Sep 14 11:24:03.186651 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.186635    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:24:03.196005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.195972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:03.823342 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.823303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" event={"ID":"d072ca32-7a4a-4a11-8e2e-be109303b686","Type":"ContainerStarted","Data":"81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715"}
Sep 14 11:24:03.823528 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.823383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" event={"ID":"d072ca32-7a4a-4a11-8e2e-be109303b686","Type":"ContainerStarted","Data":"2de199dd47cba22ceec589e722805650805367169f65299e8bf60a2e3bc632f9"}
Sep 14 11:24:03.823528 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.823427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:03.844003 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:03.843950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" podStartSLOduration=2.423630891 podStartE2EDuration="2.8439325s" totalImagesPullingTime="420.301609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:03.842345486 +0000 UTC m=+1392.464898710" watchObservedRunningTime="2026-09-14 11:24:03.8439325 +0000 UTC m=+1392.466485757"
Sep 14 11:24:04.106475 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:04.106434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:05.010504 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:05.010459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:05.749453 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:05.749385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:05.912499 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:05.912459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:09.255852 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:09.255817    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"]
Sep 14 11:24:09.291531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:09.291490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:09.836016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:09.835980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:24:20.385413 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:20.385350    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:24:23.088424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:23.088370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:24.275899 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.275858    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22" podUID="d38a6f59-ec08-424a-a065-8b6e2d76ab1c" containerName="main" containerID="cri-o://6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332" gracePeriod=45
Sep 14 11:24:24.528078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.528010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:24:24.623220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-dshm\") pod \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") "
Sep 14 11:24:24.623220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfj25\" (UniqueName: \"kubernetes.io/projected/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-kube-api-access-wfj25\") pod \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") "
Sep 14 11:24:24.623531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tls-certs\") pod \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") "
Sep 14 11:24:24.623531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-home\") pod \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") "
Sep 14 11:24:24.623531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-model-cache\") pod \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") "
Sep 14 11:24:24.623698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623582    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tmp-dir\") pod \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\" (UID: \"d38a6f59-ec08-424a-a065-8b6e2d76ab1c\") "
Sep 14 11:24:24.623698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-home" (OuterVolumeSpecName: "home") pod "d38a6f59-ec08-424a-a065-8b6e2d76ab1c" (UID: "d38a6f59-ec08-424a-a065-8b6e2d76ab1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:24.623808 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d38a6f59-ec08-424a-a065-8b6e2d76ab1c" (UID: "d38a6f59-ec08-424a-a065-8b6e2d76ab1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:24.623915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d38a6f59-ec08-424a-a065-8b6e2d76ab1c" (UID: "d38a6f59-ec08-424a-a065-8b6e2d76ab1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:24.624016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.623999    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.624068 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.624022    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.624068 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.624036    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.625649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.625625    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-kube-api-access-wfj25" (OuterVolumeSpecName: "kube-api-access-wfj25") pod "d38a6f59-ec08-424a-a065-8b6e2d76ab1c" (UID: "d38a6f59-ec08-424a-a065-8b6e2d76ab1c"). InnerVolumeSpecName "kube-api-access-wfj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:24:24.625898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.625882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-dshm" (OuterVolumeSpecName: "dshm") pod "d38a6f59-ec08-424a-a065-8b6e2d76ab1c" (UID: "d38a6f59-ec08-424a-a065-8b6e2d76ab1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:24:24.625956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.625898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d38a6f59-ec08-424a-a065-8b6e2d76ab1c" (UID: "d38a6f59-ec08-424a-a065-8b6e2d76ab1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:24:24.724965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.724916    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.724965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.724954    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfj25\" (UniqueName: \"kubernetes.io/projected/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-kube-api-access-wfj25\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.724965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.724964    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38a6f59-ec08-424a-a065-8b6e2d76ab1c-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:24:24.954512 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.954479    2563 generic.go:356] "Generic (PLEG): container finished" podID="d38a6f59-ec08-424a-a065-8b6e2d76ab1c" containerID="6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332" exitCode=0
Sep 14 11:24:24.954708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.954530    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"
Sep 14 11:24:24.954708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.954569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22" event={"ID":"d38a6f59-ec08-424a-a065-8b6e2d76ab1c","Type":"ContainerDied","Data":"6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332"}
Sep 14 11:24:24.954708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.954617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22" event={"ID":"d38a6f59-ec08-424a-a065-8b6e2d76ab1c","Type":"ContainerDied","Data":"6214436a139f220a154434b0d2679ee1557e680b63de0d50f32b2fd89c133801"}
Sep 14 11:24:24.954708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.954638    2563 scope.go:117] "RemoveContainer" containerID="6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332"
Sep 14 11:24:24.965115 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.965089    2563 scope.go:117] "RemoveContainer" containerID="6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332"
Sep 14 11:24:24.965461 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:24:24.965441    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332\": container with ID starting with 6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332 not found: ID does not exist" containerID="6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332"
Sep 14 11:24:24.965515 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.965472    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332"} err="failed to get container status \"6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332\": rpc error: code = NotFound desc = could not find container \"6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332\": container with ID starting with 6e1044b8866367c65247a887395962541fccf37e1f3c625fdb2d2ac484914332 not found: ID does not exist"
Sep 14 11:24:24.987039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.987003    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"]
Sep 14 11:24:24.990958 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:24.990930    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-b-921ff281-kserve-cccc85f4-6ph22"]
Sep 14 11:24:25.046643 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:25.046603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:25.919033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:25.918987    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38a6f59-ec08-424a-a065-8b6e2d76ab1c" path="/var/lib/kubelet/pods/d38a6f59-ec08-424a-a065-8b6e2d76ab1c/volumes"
Sep 14 11:24:26.008473 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:26.008440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:27.367942 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:27.367898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:28.343602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:28.343563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:28.751336 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:28.751298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:29.125953 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:29.125901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:29.428545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:29.428452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:29.751849 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:29.751754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:29.893294 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:29.893259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:30.789268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:30.789226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:31.896739 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:31.896698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:33.135076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:33.134743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:33.274722 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:33.274680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:33.655896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:33.655858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:34.753214 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:34.753172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:35.747795 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:35.747761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:36.558107 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.558065    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"]
Sep 14 11:24:36.558736 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.558715    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38a6f59-ec08-424a-a065-8b6e2d76ab1c" containerName="main"
Sep 14 11:24:36.558821 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.558742    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38a6f59-ec08-424a-a065-8b6e2d76ab1c" containerName="main"
Sep 14 11:24:36.558898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.558886    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38a6f59-ec08-424a-a065-8b6e2d76ab1c" containerName="main"
Sep 14 11:24:36.563354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.563331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.575786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.575764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7cc544b6\"/\"kube-root-ca.crt\""
Sep 14 11:24:36.585912 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.585887    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7cc544b6\"/\"default-dockercfg-znmk8\""
Sep 14 11:24:36.593106 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.593085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7cc544b6\"/\"e2e-lifecycle-model-7cc544b6-kserve-self-signed-certs\""
Sep 14 11:24:36.596971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.596948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7cc544b6\"/\"openshift-service-ca.crt\""
Sep 14 11:24:36.632670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.632639    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"]
Sep 14 11:24:36.641731 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.641680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-home\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.641885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.641816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tmp-dir\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.641885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.641858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-model-cache\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.641960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.641939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whbt\" (UniqueName: \"kubernetes.io/projected/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-kube-api-access-5whbt\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.642001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.641972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-dshm\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.642001 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.641996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tls-certs\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.660006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.659958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:36.743312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-model-cache\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whbt\" (UniqueName: \"kubernetes.io/projected/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-kube-api-access-5whbt\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-dshm\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tls-certs\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743546 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-home\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743781 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tmp-dir\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743781 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-model-cache\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tmp-dir\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.743972 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.743941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-home\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.745714 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.745693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-dshm\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.746005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.745985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tls-certs\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.767221 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.767192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whbt\" (UniqueName: \"kubernetes.io/projected/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-kube-api-access-5whbt\") pod \"e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") " pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:36.872568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:36.872528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:37.155832 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:37.155799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"]
Sep 14 11:24:37.156974 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:24:37.156941    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc85e3ec9_10fd_44f2_ab4d_d64f45e6599b.slice/crio-2e2b50827f2defc74c94a909d8d867382d3cd3b1b266584b56c0ed14d6eda729 WatchSource:0}: Error finding container 2e2b50827f2defc74c94a909d8d867382d3cd3b1b266584b56c0ed14d6eda729: Status 404 returned error can't find the container with id 2e2b50827f2defc74c94a909d8d867382d3cd3b1b266584b56c0ed14d6eda729
Sep 14 11:24:37.179746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:37.179699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:38.049955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:38.049891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm" event={"ID":"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b","Type":"ContainerStarted","Data":"12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6"}
Sep 14 11:24:38.049955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:38.049957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm" event={"ID":"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b","Type":"ContainerStarted","Data":"2e2b50827f2defc74c94a909d8d867382d3cd3b1b266584b56c0ed14d6eda729"}
Sep 14 11:24:38.121247 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:38.121194    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm" podStartSLOduration=2.121177448 podStartE2EDuration="2.121177448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:38.116674604 +0000 UTC m=+1426.739227817" watchObservedRunningTime="2026-09-14 11:24:38.121177448 +0000 UTC m=+1426.743730662"
Sep 14 11:24:38.485333 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:38.485298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:39.390610 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:39.390569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:40.225840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:40.225809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:40.975130 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:40.975091    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"]
Sep 14 11:24:40.979580 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:40.979553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:40.986542 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:40.986512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3f02a5-serving-cert\""
Sep 14 11:24:40.999459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:40.999430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"]
Sep 14 11:24:41.029676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.029634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:41.087862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.087823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.088056 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.087878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.088056 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.087913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lb2\" (UniqueName: \"kubernetes.io/projected/0e51aaea-b2cc-495b-bf46-212c2d8112d6-kube-api-access-f8lb2\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.088149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.088049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e51aaea-b2cc-495b-bf46-212c2d8112d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.130838 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.130800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:41.188689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.188646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.188900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.188701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lb2\" (UniqueName: \"kubernetes.io/projected/0e51aaea-b2cc-495b-bf46-212c2d8112d6-kube-api-access-f8lb2\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.188900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.188762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e51aaea-b2cc-495b-bf46-212c2d8112d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.188900 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:24:41.188805    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-serving-cert: secret "maas-api-e2e-shared-b-w3-3f02a5-serving-cert" not found
Sep 14 11:24:41.188900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.188812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.188900 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:24:41.188890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls podName:0e51aaea-b2cc-495b-bf46-212c2d8112d6 nodeName:}" failed. No retries permitted until 2026-09-14 11:24:41.688867709 +0000 UTC m=+1430.311420900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls") pod "maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" (UID: "0e51aaea-b2cc-495b-bf46-212c2d8112d6") : secret "maas-api-e2e-shared-b-w3-3f02a5-serving-cert" not found
Sep 14 11:24:41.189530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.189504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e51aaea-b2cc-495b-bf46-212c2d8112d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.191354 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.191333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.198299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.198274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lb2\" (UniqueName: \"kubernetes.io/projected/0e51aaea-b2cc-495b-bf46-212c2d8112d6-kube-api-access-f8lb2\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.289090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.288997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:41.692899 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.692851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.695293 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.695259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:41.706294 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.706264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:41.808076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.808039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:41.892702 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:41.892657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:42.174837 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:42.174808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"]
Sep 14 11:24:42.175893 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:24:42.175868    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e51aaea_b2cc_495b_bf46_212c2d8112d6.slice/crio-9170f14e42bb21b2c595c4bc47fc7b92f7a03d3c85a773c84ef2aa82b2b90dfb WatchSource:0}: Error finding container 9170f14e42bb21b2c595c4bc47fc7b92f7a03d3c85a773c84ef2aa82b2b90dfb: Status 404 returned error can't find the container with id 9170f14e42bb21b2c595c4bc47fc7b92f7a03d3c85a773c84ef2aa82b2b90dfb
Sep 14 11:24:43.082020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:43.081908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" event={"ID":"0e51aaea-b2cc-495b-bf46-212c2d8112d6","Type":"ContainerStarted","Data":"0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c"}
Sep 14 11:24:43.082020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:43.081970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" event={"ID":"0e51aaea-b2cc-495b-bf46-212c2d8112d6","Type":"ContainerStarted","Data":"9170f14e42bb21b2c595c4bc47fc7b92f7a03d3c85a773c84ef2aa82b2b90dfb"}
Sep 14 11:24:43.082020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:43.081995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:43.118750 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:43.118697    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" podStartSLOduration=2.638245265 podStartE2EDuration="3.118680349s" totalImagesPullingTime="480.435084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:43.118177163 +0000 UTC m=+1431.740730378" watchObservedRunningTime="2026-09-14 11:24:43.118680349 +0000 UTC m=+1431.741233563"
Sep 14 11:24:46.873078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:46.872971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:46.873078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:46.873037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:46.885775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:46.885744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:47.117277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:47.117247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:24:49.093661 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:49.093632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:24:54.658740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:54.658697    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:24:56.108664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:56.108627    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"]
Sep 14 11:24:56.141771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:56.141719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:58.043899 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:58.043866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:24:59.452860 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:24:59.452826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:04.153697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:04.153643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:05.062234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:05.062189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:05.746114 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:05.746079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:06.603633 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:06.603604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:07.545154 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:07.545097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:08.125420 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:08.125354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:08.581942 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:08.581854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:09.767531 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:09.767490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:10.762184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:10.762145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:11.129426 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.129360    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6" podUID="f2ca1660-436c-46b4-b914-d5c936a987f9" containerName="main" containerID="cri-o://64e77723f98497a58aa8bb2ca83892f4b7ee4fe2ef66a5f5b5a548b2aa9ddd04" gracePeriod=45
Sep 14 11:25:11.257187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.257154    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2ca1660-436c-46b4-b914-d5c936a987f9" containerID="64e77723f98497a58aa8bb2ca83892f4b7ee4fe2ef66a5f5b5a548b2aa9ddd04" exitCode=0
Sep 14 11:25:11.257382 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.257222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6" event={"ID":"f2ca1660-436c-46b4-b914-d5c936a987f9","Type":"ContainerDied","Data":"64e77723f98497a58aa8bb2ca83892f4b7ee4fe2ef66a5f5b5a548b2aa9ddd04"}
Sep 14 11:25:11.391888 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.391816    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:25:11.492977 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.492937    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-model-cache\") pod \"f2ca1660-436c-46b4-b914-d5c936a987f9\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") "
Sep 14 11:25:11.493181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.492990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shhrn\" (UniqueName: \"kubernetes.io/projected/f2ca1660-436c-46b4-b914-d5c936a987f9-kube-api-access-shhrn\") pod \"f2ca1660-436c-46b4-b914-d5c936a987f9\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") "
Sep 14 11:25:11.493181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca1660-436c-46b4-b914-d5c936a987f9-tls-certs\") pod \"f2ca1660-436c-46b4-b914-d5c936a987f9\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") "
Sep 14 11:25:11.493181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-dshm\") pod \"f2ca1660-436c-46b4-b914-d5c936a987f9\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") "
Sep 14 11:25:11.493181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-tmp-dir\") pod \"f2ca1660-436c-46b4-b914-d5c936a987f9\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") "
Sep 14 11:25:11.493181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-home\") pod \"f2ca1660-436c-46b4-b914-d5c936a987f9\" (UID: \"f2ca1660-436c-46b4-b914-d5c936a987f9\") "
Sep 14 11:25:11.493484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2ca1660-436c-46b4-b914-d5c936a987f9" (UID: "f2ca1660-436c-46b4-b914-d5c936a987f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:25:11.493484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2ca1660-436c-46b4-b914-d5c936a987f9" (UID: "f2ca1660-436c-46b4-b914-d5c936a987f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:25:11.493567 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-home" (OuterVolumeSpecName: "home") pod "f2ca1660-436c-46b4-b914-d5c936a987f9" (UID: "f2ca1660-436c-46b4-b914-d5c936a987f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:25:11.493567 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493540    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:25:11.493567 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.493558    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:25:11.495214 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.495192    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-dshm" (OuterVolumeSpecName: "dshm") pod "f2ca1660-436c-46b4-b914-d5c936a987f9" (UID: "f2ca1660-436c-46b4-b914-d5c936a987f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:25:11.495681 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.495655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ca1660-436c-46b4-b914-d5c936a987f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2ca1660-436c-46b4-b914-d5c936a987f9" (UID: "f2ca1660-436c-46b4-b914-d5c936a987f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:25:11.495783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.495658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ca1660-436c-46b4-b914-d5c936a987f9-kube-api-access-shhrn" (OuterVolumeSpecName: "kube-api-access-shhrn") pod "f2ca1660-436c-46b4-b914-d5c936a987f9" (UID: "f2ca1660-436c-46b4-b914-d5c936a987f9"). InnerVolumeSpecName "kube-api-access-shhrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:25:11.595046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.595002    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhrn\" (UniqueName: \"kubernetes.io/projected/f2ca1660-436c-46b4-b914-d5c936a987f9-kube-api-access-shhrn\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:25:11.595046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.595040    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca1660-436c-46b4-b914-d5c936a987f9-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:25:11.595046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.595052    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:25:11.595287 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:11.595061    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca1660-436c-46b4-b914-d5c936a987f9-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:25:12.266224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:12.266130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"
Sep 14 11:25:12.266224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:12.266172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6" event={"ID":"f2ca1660-436c-46b4-b914-d5c936a987f9","Type":"ContainerDied","Data":"5d96e8e1b5a926571fb44380382821911478f6db1d83397bd80436a8ce69f66e"}
Sep 14 11:25:12.266224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:12.266224    2563 scope.go:117] "RemoveContainer" containerID="64e77723f98497a58aa8bb2ca83892f4b7ee4fe2ef66a5f5b5a548b2aa9ddd04"
Sep 14 11:25:12.284950 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:12.284914    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"]
Sep 14 11:25:12.292070 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:12.292037    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/e2e-conflict-a-921ff281-kserve-6999758f56-xhdx6"]
Sep 14 11:25:13.920010 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:13.919975    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ca1660-436c-46b4-b914-d5c936a987f9" path="/var/lib/kubelet/pods/f2ca1660-436c-46b4-b914-d5c936a987f9/volumes"
Sep 14 11:25:27.237099 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.237059    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"]
Sep 14 11:25:27.237784 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.237762    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ca1660-436c-46b4-b914-d5c936a987f9" containerName="main"
Sep 14 11:25:27.237836 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.237788    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ca1660-436c-46b4-b914-d5c936a987f9" containerName="main"
Sep 14 11:25:27.237934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.237922    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ca1660-436c-46b4-b914-d5c936a987f9" containerName="main"
Sep 14 11:25:27.241319 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.241300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.243729 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.243705    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-de0f7a\"/\"distinct-w4-de0f7a-kserve-self-signed-certs\""
Sep 14 11:25:27.251657 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.251628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"]
Sep 14 11:25:27.267850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.267809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:27.354430 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.354351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tls-certs\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.354430 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.354435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-model-cache\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.354706 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.354475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tmp-dir\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.354706 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.354513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnmm\" (UniqueName: \"kubernetes.io/projected/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-kube-api-access-5cnmm\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.354706 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.354595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-home\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.354706 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.354630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-dshm\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.455647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.455611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-home\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.455833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.455654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-dshm\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.455833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.455719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tls-certs\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.455833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.455746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-model-cache\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.455833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.455801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tmp-dir\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.456038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.455839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnmm\" (UniqueName: \"kubernetes.io/projected/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-kube-api-access-5cnmm\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.456129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.456104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-home\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.456201 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.456179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-model-cache\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.456307 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.456284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tmp-dir\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.458038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.458015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-dshm\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.458417 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.458380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tls-certs\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.464628 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.464605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnmm\" (UniqueName: \"kubernetes.io/projected/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-kube-api-access-5cnmm\") pod \"distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") " pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.553467 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.553363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:27.832619 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:27.832591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"]
Sep 14 11:25:27.834285 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:25:27.834257    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5a4e4ef_93df_43e5_bd63_0c02fb987c26.slice/crio-097fe895c352546c4e4b081da3a550aef867c866f69c9af5e164f2ae5ad22b3d WatchSource:0}: Error finding container 097fe895c352546c4e4b081da3a550aef867c866f69c9af5e164f2ae5ad22b3d: Status 404 returned error can't find the container with id 097fe895c352546c4e4b081da3a550aef867c866f69c9af5e164f2ae5ad22b3d
Sep 14 11:25:28.369704 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:28.369653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z" event={"ID":"c5a4e4ef-93df-43e5-bd63-0c02fb987c26","Type":"ContainerStarted","Data":"781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a"}
Sep 14 11:25:28.370114 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:28.369721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z" event={"ID":"c5a4e4ef-93df-43e5-bd63-0c02fb987c26","Type":"ContainerStarted","Data":"097fe895c352546c4e4b081da3a550aef867c866f69c9af5e164f2ae5ad22b3d"}
Sep 14 11:25:28.399223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:28.399164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z" podStartSLOduration=1.399144255 podStartE2EDuration="1.399144255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:25:28.39854318 +0000 UTC m=+1477.021096399" watchObservedRunningTime="2026-09-14 11:25:28.399144255 +0000 UTC m=+1477.021697469"
Sep 14 11:25:29.868567 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:29.868534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:25:34.088130 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:34.088079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:35.004666 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:35.004625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:35.782075 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:35.782042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:36.515151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:36.515115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:37.334003 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:37.333920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:37.554366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:37.554319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:37.554605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:37.554417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:37.567432 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:37.567381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:38.069270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.069235    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"]
Sep 14 11:25:38.073108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.073088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.077427 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.077402    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"default-dockercfg-tfffw\""
Sep 14 11:25:38.077427 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.077421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"kube-root-ca.crt\""
Sep 14 11:25:38.077642 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.077430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"auth-test-model-55c025-kserve-self-signed-certs\""
Sep 14 11:25:38.077642 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.077479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"openshift-service-ca.crt\""
Sep 14 11:25:38.087175 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.087146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"]
Sep 14 11:25:38.094525 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.094495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:38.164938 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.164900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-tmp-dir\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.165158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.164944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d42808c-99de-4d62-9e31-51bb9f29ff87-tls-certs\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.165158 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.165053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-model-cache\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.165268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.165158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-dshm\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.165268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.165197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-home\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.165268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.165216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgtt\" (UniqueName: \"kubernetes.io/projected/2d42808c-99de-4d62-9e31-51bb9f29ff87-kube-api-access-flgtt\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.192869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.192833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:38.266289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-dshm\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-home\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgtt\" (UniqueName: \"kubernetes.io/projected/2d42808c-99de-4d62-9e31-51bb9f29ff87-kube-api-access-flgtt\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-tmp-dir\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d42808c-99de-4d62-9e31-51bb9f29ff87-tls-certs\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-model-cache\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266815 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-home\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-tmp-dir\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.266898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.266883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-model-cache\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.268913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.268881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-dshm\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.269127 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.269109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d42808c-99de-4d62-9e31-51bb9f29ff87-tls-certs\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.274792 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.274771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgtt\" (UniqueName: \"kubernetes.io/projected/2d42808c-99de-4d62-9e31-51bb9f29ff87-kube-api-access-flgtt\") pod \"auth-test-model-55c025-kserve-54c6679db7-5qc9s\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") " pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.383646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.383608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:38.447658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.447599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:25:38.662269 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:38.662237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"]
Sep 14 11:25:38.663403 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:25:38.663369    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d42808c_99de_4d62_9e31_51bb9f29ff87.slice/crio-1c68c4857faa486211f117018b46b28d1804e1e0899d2597b9650c1e4077d411 WatchSource:0}: Error finding container 1c68c4857faa486211f117018b46b28d1804e1e0899d2597b9650c1e4077d411: Status 404 returned error can't find the container with id 1c68c4857faa486211f117018b46b28d1804e1e0899d2597b9650c1e4077d411
Sep 14 11:25:39.077461 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:39.077281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:39.439577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:39.439532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s" event={"ID":"2d42808c-99de-4d62-9e31-51bb9f29ff87","Type":"ContainerStarted","Data":"5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1"}
Sep 14 11:25:39.439577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:39.439582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s" event={"ID":"2d42808c-99de-4d62-9e31-51bb9f29ff87","Type":"ContainerStarted","Data":"1c68c4857faa486211f117018b46b28d1804e1e0899d2597b9650c1e4077d411"}
Sep 14 11:25:39.458440 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:39.458365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s" podStartSLOduration=1.458347903 podStartE2EDuration="1.458347903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:25:39.457464704 +0000 UTC m=+1488.080017924" watchObservedRunningTime="2026-09-14 11:25:39.458347903 +0000 UTC m=+1488.080901117"
Sep 14 11:25:39.805916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:39.805830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:40.703622 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:40.703587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:48.384305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:48.384267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:48.384917 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:48.384324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:48.397355 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:48.397324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:48.509790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:48.509756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:25:49.365940 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:49.365899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:49.442288 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:49.442255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:49.544700 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:49.544664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:49.629366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:49.629274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:49.700955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:49.700915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:50.090819 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:50.090729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:51.021326 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:51.021286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:51.824621 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:51.824579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:51.979697 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:51.979668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:25:51.991998 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:51.991973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:25:52.068223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:52.068189    2563 scope.go:117] "RemoveContainer" containerID="2c9a3ba756767ba228cccfa34f93a9133589c1681aa495121692820af8d5206c"
Sep 14 11:25:52.622730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:52.622694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:52.719708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:52.719677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:53.529165 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:53.529124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:54.286964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:54.286929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:54.460951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:54.460913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:55.196930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:55.196891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:25:57.601597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:57.601548    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"]
Sep 14 11:25:57.657595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:25:57.657557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:05.645315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:05.645274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:07.150982 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:07.150918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:26:09.091654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:09.091614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:10.042241 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:10.042195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:12.621675 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:12.621631    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm" podUID="c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" containerName="main" containerID="cri-o://12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6" gracePeriod=45
Sep 14 11:26:12.884064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:12.883991    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:26:13.020691 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tmp-dir\") pod \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") "
Sep 14 11:26:13.020850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-model-cache\") pod \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") "
Sep 14 11:26:13.020850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-dshm\") pod \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") "
Sep 14 11:26:13.020850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-home\") pod \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") "
Sep 14 11:26:13.020850 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5whbt\" (UniqueName: \"kubernetes.io/projected/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-kube-api-access-5whbt\") pod \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") "
Sep 14 11:26:13.021049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tls-certs\") pod \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\" (UID: \"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b\") "
Sep 14 11:26:13.021049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.020971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" (UID: "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:26:13.021163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.021053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" (UID: "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:26:13.021163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.021062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-home" (OuterVolumeSpecName: "home") pod "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" (UID: "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:26:13.021280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.021241    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:26:13.021280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.021256    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:26:13.021280 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.021267    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:26:13.023052 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.023024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-dshm" (OuterVolumeSpecName: "dshm") pod "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" (UID: "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:26:13.023187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.023131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-kube-api-access-5whbt" (OuterVolumeSpecName: "kube-api-access-5whbt") pod "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" (UID: "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b"). InnerVolumeSpecName "kube-api-access-5whbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:26:13.023187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.023164    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" (UID: "c85e3ec9-10fd-44f2-ab4d-d64f45e6599b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:26:13.122229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.122183    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:26:13.122229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.122223    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whbt\" (UniqueName: \"kubernetes.io/projected/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-kube-api-access-5whbt\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:26:13.122229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.122237    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:26:13.673073 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.673037    2563 generic.go:356] "Generic (PLEG): container finished" podID="c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" containerID="12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6" exitCode=0
Sep 14 11:26:13.673506 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.673085    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"
Sep 14 11:26:13.673506 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.673118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm" event={"ID":"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b","Type":"ContainerDied","Data":"12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6"}
Sep 14 11:26:13.673506 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.673167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm" event={"ID":"c85e3ec9-10fd-44f2-ab4d-d64f45e6599b","Type":"ContainerDied","Data":"2e2b50827f2defc74c94a909d8d867382d3cd3b1b266584b56c0ed14d6eda729"}
Sep 14 11:26:13.673506 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.673184    2563 scope.go:117] "RemoveContainer" containerID="12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6"
Sep 14 11:26:13.683594 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.683572    2563 scope.go:117] "RemoveContainer" containerID="12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6"
Sep 14 11:26:13.683886 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:26:13.683862    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6\": container with ID starting with 12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6 not found: ID does not exist" containerID="12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6"
Sep 14 11:26:13.683959 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.683893    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6"} err="failed to get container status \"12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6\": rpc error: code = NotFound desc = could not find container \"12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6\": container with ID starting with 12e7ddb995a5e77c7dee21b917ae417aad8e2b2c03e8d7fe6fa039340be942b6 not found: ID does not exist"
Sep 14 11:26:13.696408 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.696357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"]
Sep 14 11:26:13.700459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.700420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7cc544b6/e2e-lifecycle-model-7cc544b6-kserve-7b648944b7-twpxm"]
Sep 14 11:26:13.919412 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:13.919361    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" path="/var/lib/kubelet/pods/c85e3ec9-10fd-44f2-ab4d-d64f45e6599b/volumes"
Sep 14 11:26:30.240945 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:30.240900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:30.605153 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:30.605117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:33.944923 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:33.944884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:34.941641 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:34.941604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:35.603771 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:35.603729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:37.967808 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:37.967764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:26:41.923144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:41.923108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:48.634135 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.634090    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"]
Sep 14 11:26:48.634777 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.634754    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" containerName="main"
Sep 14 11:26:48.634870 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.634779    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" containerName="main"
Sep 14 11:26:48.634924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.634891    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85e3ec9-10fd-44f2-ab4d-d64f45e6599b" containerName="main"
Sep 14 11:26:48.637875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.637850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.640999 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.640965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"auth-test-model-3f02a5-kserve-self-signed-certs\""
Sep 14 11:26:48.641150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.640983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"default-dockercfg-q828q\""
Sep 14 11:26:48.641150 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.641032    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"openshift-service-ca.crt\""
Sep 14 11:26:48.641326 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.640981    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"kube-root-ca.crt\""
Sep 14 11:26:48.652002 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.651971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"]
Sep 14 11:26:48.675441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.675373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:48.679435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.679407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-model-cache\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.679589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.679441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297h7\" (UniqueName: \"kubernetes.io/projected/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-kube-api-access-297h7\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.679589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.679511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-dshm\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.679589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.679533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tls-certs\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.679699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.679585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-home\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.679699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.679655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tmp-dir\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781126 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-model-cache\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781126 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297h7\" (UniqueName: \"kubernetes.io/projected/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-kube-api-access-297h7\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-dshm\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tls-certs\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-home\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tmp-dir\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781626 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-model-cache\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-home\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.781774 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.781725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tmp-dir\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.783555 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.783535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-dshm\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.783805 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.783790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tls-certs\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.790177 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.790156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-297h7\" (UniqueName: \"kubernetes.io/projected/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-kube-api-access-297h7\") pod \"auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:48.948340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:48.948251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:49.226163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:49.226136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"]
Sep 14 11:26:49.228145 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:26:49.228120    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53d1a20e_f89c_4b3e_9f94_8d51c59eeeb4.slice/crio-afcc105932f6ee3042d49925da8edead9a97456e29dc942bfc31ec4d8407a965 WatchSource:0}: Error finding container afcc105932f6ee3042d49925da8edead9a97456e29dc942bfc31ec4d8407a965: Status 404 returned error can't find the container with id afcc105932f6ee3042d49925da8edead9a97456e29dc942bfc31ec4d8407a965
Sep 14 11:26:49.902258 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:49.902216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p" event={"ID":"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4","Type":"ContainerStarted","Data":"f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3"}
Sep 14 11:26:49.902692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:49.902269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p" event={"ID":"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4","Type":"ContainerStarted","Data":"afcc105932f6ee3042d49925da8edead9a97456e29dc942bfc31ec4d8407a965"}
Sep 14 11:26:49.924325 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:49.924264    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p" podStartSLOduration=1.924249218 podStartE2EDuration="1.924249218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:26:49.921366603 +0000 UTC m=+1558.543919819" watchObservedRunningTime="2026-09-14 11:26:49.924249218 +0000 UTC m=+1558.546802433"
Sep 14 11:26:52.100744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:52.100711    2563 scope.go:117] "RemoveContainer" containerID="1e2fb9b7c4371297afae13ea75fdf96edfbf60a6f46274fe96f6ade8ffc7f3de"
Sep 14 11:26:56.583324 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:56.583270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:56.651874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:56.651837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:56.826805 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:56.826770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:58.031898 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.031863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:58.142196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.142148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:58.776058 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.776015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:58.876007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.875963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:26:58.948760 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.948720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:58.948925 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.948782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:58.962076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.962045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:58.973473 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.973441    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:26:58.975804 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:26:58.975777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:00.148638 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:00.148599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:06.409492 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:06.409453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:06.511065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:06.511029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:07.043807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:07.043768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:11.849377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:11.849340    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:27:12.955745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:12.955696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:13.020002 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:13.019965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:13.112263 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:13.112220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:13.189749 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:13.189708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:17.350807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:17.350746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:23.353945 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:23.353903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:23.724843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:23.724745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:23.862015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:23.861981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:37.772956 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:37.772914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:44.747765 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:44.747672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:45.856182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:45.856142    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:27:51.464368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:51.464331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:51.598628 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:51.598591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:51.794716 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:51.794623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:52.131737 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:52.131705    2563 scope.go:117] "RemoveContainer" containerID="3a991a9a94a84f2ab0bac4bef177f12a57a9027a49425d4a3168c0ae7287a1aa"
Sep 14 11:27:52.172113 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:52.172072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:52.558782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:52.558688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:54.385605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:54.385573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:54.987347 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:54.987312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:55.078438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:55.078383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:55.385468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:55.385427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:56.304533 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:56.304495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:57.016688 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:57.016648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:57.718131 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:57.718086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:58.635677 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:58.635642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:27:59.506519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:27:59.506484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:00.549265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:00.549230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:00.647459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:00.647410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:00.982660 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:00.982627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:01.146993 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:01.146956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:01.247187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:01.247092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:14.918797 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:14.918751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:21.636948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:21.636906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:22.984579 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:22.984542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:23.840790 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:23.840757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:25.184684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:25.184652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:25.288130 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:25.288093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:25.512370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:25.512272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:25.596629 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:25.596596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:25.788510 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:25.788400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:25.957503 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:25.957465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:26.492484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:26.492448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:27.209217 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:27.209165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:28.103851 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:28.103806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:28.851971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:28.851929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:29.181104 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:29.181008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:29.280759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:29.280725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:29.768502 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:29.768463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:30.211965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:30.211931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:30.614496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:30.614457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:31.337241 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:31.337205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:32.024051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:32.024014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:32.851369 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:32.851328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:33.523699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:33.523659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:34.289987 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:34.289941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:34.966399 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:34.966358    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"]
Sep 14 11:28:34.971076 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:34.971044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:34.973381 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:34.973348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-420d64-serving-cert\""
Sep 14 11:28:34.981328 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:34.981295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"]
Sep 14 11:28:34.997658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:34.997625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:35.058021 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.057985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:35.098639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.098601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-metrics-certs\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.098807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.098643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gjz\" (UniqueName: \"kubernetes.io/projected/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-kube-api-access-f2gjz\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.098807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.098685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-maas-api-tls\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.098807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.098776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-openshift-service-ca\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.199662 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.199628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-metrics-certs\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.199662 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.199667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gjz\" (UniqueName: \"kubernetes.io/projected/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-kube-api-access-f2gjz\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.199919 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.199699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-maas-api-tls\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.199919 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.199733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-openshift-service-ca\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.200405 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.200373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-openshift-service-ca\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.202223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.202198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-maas-api-tls\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.202320 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.202255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-metrics-certs\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.209620 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.209589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gjz\" (UniqueName: \"kubernetes.io/projected/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-kube-api-access-f2gjz\") pod \"maas-api-e2e-api-b-420d64-8695bd86b6-cc52c\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.246852 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.246758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:35.287025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.286984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:35.483712 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.483675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:35.568451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.568423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"]
Sep 14 11:28:35.569495 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:35.569465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:35.570221 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:28:35.570189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8ccdf3_9ea0_42ec_bd84_62d76140ce22.slice/crio-5d9f1cc5eb830adbcc7f3fb5fc9031192c7ad37273a1bf05dbe88eed1daf55e0 WatchSource:0}: Error finding container 5d9f1cc5eb830adbcc7f3fb5fc9031192c7ad37273a1bf05dbe88eed1daf55e0: Status 404 returned error can't find the container with id 5d9f1cc5eb830adbcc7f3fb5fc9031192c7ad37273a1bf05dbe88eed1daf55e0
Sep 14 11:28:36.563522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:36.563482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c" event={"ID":"df8ccdf3-9ea0-42ec-bd84-62d76140ce22","Type":"ContainerStarted","Data":"78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0"}
Sep 14 11:28:36.563901 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:36.563537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c" event={"ID":"df8ccdf3-9ea0-42ec-bd84-62d76140ce22","Type":"ContainerStarted","Data":"5d9f1cc5eb830adbcc7f3fb5fc9031192c7ad37273a1bf05dbe88eed1daf55e0"}
Sep 14 11:28:36.563901 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:36.563575    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:36.585643 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:36.585587    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c" podStartSLOduration=2.220126181 podStartE2EDuration="2.58557054s" totalImagesPullingTime="365.444359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:36.584587523 +0000 UTC m=+1665.207140736" watchObservedRunningTime="2026-09-14 11:28:36.58557054 +0000 UTC m=+1665.208123755"
Sep 14 11:28:38.403084 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:38.403047    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:28:42.575303 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:42.575275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:28:45.652235 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:45.652175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:46.096194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:46.096099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:47.309791 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:47.309747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:48.102164 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:48.102119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:48.896948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:48.896914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:49.860596 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:49.860554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:50.627856 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:50.627818    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"]
Sep 14 11:28:50.667422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:50.661750    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"]
Sep 14 11:28:50.761081 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:50.761045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:50.860073 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:50.860037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:51.595169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:51.595133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:52.597313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:52.597276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:53.398608 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:53.398564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:54.075803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:54.075762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:54.202263 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:54.202222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:54.951789 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:54.951752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:55.809024 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:55.808971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:56.511032 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:56.510991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:57.570479 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:57.570441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:58.487811 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:58.487757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:28:59.576033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:28:59.575999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:00.491187 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:00.491150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:01.413043 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:01.413001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:02.178109 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:02.178070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:03.130371 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:03.130334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:04.004334 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:04.004299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:04.718496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:04.718456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:05.371434 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.371367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:05.408535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.408505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"]
Sep 14 11:29:05.409057 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.409025    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw" podUID="722b86d3-0e85-4f0c-b26b-21d07884e2b6" containerName="payload-processing" containerID="cri-o://1c8895c82e87fc68f69999368831c1ab40bfe31f86367e20e11e93c238efaeea" gracePeriod=30
Sep 14 11:29:05.413156 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.413130    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"]
Sep 14 11:29:05.413578 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.413553    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x" podUID="56c8ba12-68b4-4969-ac93-93aff2b49e97" containerName="payload-pre-processing" containerID="cri-o://06640bc5ba4f399c5fb0d0a1b0a7c7e9d06cefd19a5ef6bd724469d4790090c1" gracePeriod=30
Sep 14 11:29:05.446422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.446374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:05.557683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.557642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:05.648806 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.648707    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx" podUID="b307312c-c91a-4039-a029-40d098e3e3a5" containerName="main" containerID="cri-o://4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1" gracePeriod=45
Sep 14 11:29:05.688542 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.688511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:05.688678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.688656    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch" podUID="f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" containerName="main" containerID="cri-o://6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd" gracePeriod=45
Sep 14 11:29:05.743474 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.743448    2563 generic.go:356] "Generic (PLEG): container finished" podID="56c8ba12-68b4-4969-ac93-93aff2b49e97" containerID="06640bc5ba4f399c5fb0d0a1b0a7c7e9d06cefd19a5ef6bd724469d4790090c1" exitCode=0
Sep 14 11:29:05.743818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.743522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x" event={"ID":"56c8ba12-68b4-4969-ac93-93aff2b49e97","Type":"ContainerDied","Data":"06640bc5ba4f399c5fb0d0a1b0a7c7e9d06cefd19a5ef6bd724469d4790090c1"}
Sep 14 11:29:05.745168 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.745150    2563 generic.go:356] "Generic (PLEG): container finished" podID="722b86d3-0e85-4f0c-b26b-21d07884e2b6" containerID="1c8895c82e87fc68f69999368831c1ab40bfe31f86367e20e11e93c238efaeea" exitCode=0
Sep 14 11:29:05.745266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.745223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw" event={"ID":"722b86d3-0e85-4f0c-b26b-21d07884e2b6","Type":"ContainerDied","Data":"1c8895c82e87fc68f69999368831c1ab40bfe31f86367e20e11e93c238efaeea"}
Sep 14 11:29:05.802851 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.802828    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:29:05.807501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.807475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:29:05.917686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.917617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsntg\" (UniqueName: \"kubernetes.io/projected/56c8ba12-68b4-4969-ac93-93aff2b49e97-kube-api-access-xsntg\") pod \"56c8ba12-68b4-4969-ac93-93aff2b49e97\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") "
Sep 14 11:29:05.917828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.917663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722b86d3-0e85-4f0c-b26b-21d07884e2b6-plugins-config-volume\") pod \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") "
Sep 14 11:29:05.917828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.917754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgmjr\" (UniqueName: \"kubernetes.io/projected/722b86d3-0e85-4f0c-b26b-21d07884e2b6-kube-api-access-dgmjr\") pod \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\" (UID: \"722b86d3-0e85-4f0c-b26b-21d07884e2b6\") "
Sep 14 11:29:05.917954 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.917866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c8ba12-68b4-4969-ac93-93aff2b49e97-plugins-config-volume\") pod \"56c8ba12-68b4-4969-ac93-93aff2b49e97\" (UID: \"56c8ba12-68b4-4969-ac93-93aff2b49e97\") "
Sep 14 11:29:05.918457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.918323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722b86d3-0e85-4f0c-b26b-21d07884e2b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "722b86d3-0e85-4f0c-b26b-21d07884e2b6" (UID: "722b86d3-0e85-4f0c-b26b-21d07884e2b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:29:05.918457 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.918417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c8ba12-68b4-4969-ac93-93aff2b49e97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56c8ba12-68b4-4969-ac93-93aff2b49e97" (UID: "56c8ba12-68b4-4969-ac93-93aff2b49e97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:29:05.921301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.921275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c8ba12-68b4-4969-ac93-93aff2b49e97-kube-api-access-xsntg" (OuterVolumeSpecName: "kube-api-access-xsntg") pod "56c8ba12-68b4-4969-ac93-93aff2b49e97" (UID: "56c8ba12-68b4-4969-ac93-93aff2b49e97"). InnerVolumeSpecName "kube-api-access-xsntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:29:05.922782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.922753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722b86d3-0e85-4f0c-b26b-21d07884e2b6-kube-api-access-dgmjr" (OuterVolumeSpecName: "kube-api-access-dgmjr") pod "722b86d3-0e85-4f0c-b26b-21d07884e2b6" (UID: "722b86d3-0e85-4f0c-b26b-21d07884e2b6"). InnerVolumeSpecName "kube-api-access-dgmjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:29:05.939786 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.939752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:05.947487 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.947467    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:29:05.958997 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:05.958975    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:29:06.018691 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-model-cache\") pod \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") "
Sep 14 11:29:06.018874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b307312c-c91a-4039-a029-40d098e3e3a5-tls-certs\") pod \"b307312c-c91a-4039-a029-40d098e3e3a5\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") "
Sep 14 11:29:06.018874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-tmp-dir\") pod \"b307312c-c91a-4039-a029-40d098e3e3a5\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") "
Sep 14 11:29:06.018874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-model-cache\") pod \"b307312c-c91a-4039-a029-40d098e3e3a5\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") "
Sep 14 11:29:06.018874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-dshm\") pod \"b307312c-c91a-4039-a029-40d098e3e3a5\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") "
Sep 14 11:29:06.018874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018829    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-home\") pod \"b307312c-c91a-4039-a029-40d098e3e3a5\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") "
Sep 14 11:29:06.018874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-dshm\") pod \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") "
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4k5l\" (UniqueName: \"kubernetes.io/projected/b307312c-c91a-4039-a029-40d098e3e3a5-kube-api-access-n4k5l\") pod \"b307312c-c91a-4039-a029-40d098e3e3a5\" (UID: \"b307312c-c91a-4039-a029-40d098e3e3a5\") "
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tmp-dir\") pod \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") "
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w2p9\" (UniqueName: \"kubernetes.io/projected/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-kube-api-access-4w2p9\") pod \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") "
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.018976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-model-cache" (OuterVolumeSpecName: "model-cache") pod "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" (UID: "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019001    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-home\") pod \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") "
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019061    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b307312c-c91a-4039-a029-40d098e3e3a5" (UID: "b307312c-c91a-4039-a029-40d098e3e3a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b307312c-c91a-4039-a029-40d098e3e3a5" (UID: "b307312c-c91a-4039-a029-40d098e3e3a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.019192 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tls-certs\") pod \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\" (UID: \"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64\") "
Sep 14 11:29:06.019646 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-home" (OuterVolumeSpecName: "home") pod "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" (UID: "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.019705 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019691    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsntg\" (UniqueName: \"kubernetes.io/projected/56c8ba12-68b4-4969-ac93-93aff2b49e97-kube-api-access-xsntg\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019712    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722b86d3-0e85-4f0c-b26b-21d07884e2b6-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019726    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmjr\" (UniqueName: \"kubernetes.io/projected/722b86d3-0e85-4f0c-b26b-21d07884e2b6-kube-api-access-dgmjr\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019742    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019754    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019966 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019767    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56c8ba12-68b4-4969-ac93-93aff2b49e97-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019966 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019779    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019966 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019791    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.019966 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.019903    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" (UID: "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.020437 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.020098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-home" (OuterVolumeSpecName: "home") pod "b307312c-c91a-4039-a029-40d098e3e3a5" (UID: "b307312c-c91a-4039-a029-40d098e3e3a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.021743 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.021700    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" (UID: "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:29:06.021840 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.021740    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b307312c-c91a-4039-a029-40d098e3e3a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b307312c-c91a-4039-a029-40d098e3e3a5" (UID: "b307312c-c91a-4039-a029-40d098e3e3a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:29:06.022261 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.022243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-kube-api-access-4w2p9" (OuterVolumeSpecName: "kube-api-access-4w2p9") pod "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" (UID: "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64"). InnerVolumeSpecName "kube-api-access-4w2p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:29:06.022779 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.022756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-dshm" (OuterVolumeSpecName: "dshm") pod "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" (UID: "f002c3af-b9f8-420a-8a8f-f6b7c05b0e64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.022937 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.022897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b307312c-c91a-4039-a029-40d098e3e3a5-kube-api-access-n4k5l" (OuterVolumeSpecName: "kube-api-access-n4k5l") pod "b307312c-c91a-4039-a029-40d098e3e3a5" (UID: "b307312c-c91a-4039-a029-40d098e3e3a5"). InnerVolumeSpecName "kube-api-access-n4k5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:29:06.023061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.022958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-dshm" (OuterVolumeSpecName: "dshm") pod "b307312c-c91a-4039-a029-40d098e3e3a5" (UID: "b307312c-c91a-4039-a029-40d098e3e3a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:29:06.040629 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.040599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:06.121360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121329    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b307312c-c91a-4039-a029-40d098e3e3a5-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121354    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121363    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b307312c-c91a-4039-a029-40d098e3e3a5-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121372    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121381    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k5l\" (UniqueName: \"kubernetes.io/projected/b307312c-c91a-4039-a029-40d098e3e3a5-kube-api-access-n4k5l\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121418    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121429    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w2p9\" (UniqueName: \"kubernetes.io/projected/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-kube-api-access-4w2p9\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.121591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.121436    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:06.752720 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.752625    2563 generic.go:356] "Generic (PLEG): container finished" podID="f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" containerID="6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd" exitCode=0
Sep 14 11:29:06.752720 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.752671    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"
Sep 14 11:29:06.752720 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.752715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch" event={"ID":"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64","Type":"ContainerDied","Data":"6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd"}
Sep 14 11:29:06.753246 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.752762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch" event={"ID":"f002c3af-b9f8-420a-8a8f-f6b7c05b0e64","Type":"ContainerDied","Data":"25d5e2e661590bb71d3eb48746749a6640deb18539122c9fa7c7e166285649e8"}
Sep 14 11:29:06.753246 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.752787    2563 scope.go:117] "RemoveContainer" containerID="6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd"
Sep 14 11:29:06.754313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.754180    2563 generic.go:356] "Generic (PLEG): container finished" podID="b307312c-c91a-4039-a029-40d098e3e3a5" containerID="4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1" exitCode=0
Sep 14 11:29:06.754313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.754223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"
Sep 14 11:29:06.754313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.754263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx" event={"ID":"b307312c-c91a-4039-a029-40d098e3e3a5","Type":"ContainerDied","Data":"4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1"}
Sep 14 11:29:06.754313 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.754302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx" event={"ID":"b307312c-c91a-4039-a029-40d098e3e3a5","Type":"ContainerDied","Data":"d6eae5cc6d141e34289f0ff668096c4b3d38d42792f28f49469fe38c3cbeb85e"}
Sep 14 11:29:06.755903 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.755873    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"
Sep 14 11:29:06.756029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.755934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw" event={"ID":"722b86d3-0e85-4f0c-b26b-21d07884e2b6","Type":"ContainerDied","Data":"fdcaaa2e2412180c9cb6a59499bc7a044d353341b99ecc351241f8abf258f1c7"}
Sep 14 11:29:06.757684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.757652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x" event={"ID":"56c8ba12-68b4-4969-ac93-93aff2b49e97","Type":"ContainerDied","Data":"b3c6c75f00071c8b2120803b9e05d43e4e443d51b42466bfb3bc63b52dff17dd"}
Sep 14 11:29:06.757684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.757685    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"
Sep 14 11:29:06.765760 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.765737    2563 scope.go:117] "RemoveContainer" containerID="6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd"
Sep 14 11:29:06.766772 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:29:06.766591    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd\": container with ID starting with 6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd not found: ID does not exist" containerID="6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd"
Sep 14 11:29:06.766772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.766658    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd"} err="failed to get container status \"6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd\": rpc error: code = NotFound desc = could not find container \"6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd\": container with ID starting with 6220e151207e4f6eaa241168d67eda43b57927b81e5a568308f7dc82ef1e8dcd not found: ID does not exist"
Sep 14 11:29:06.766772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.766683    2563 scope.go:117] "RemoveContainer" containerID="4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1"
Sep 14 11:29:06.776288 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.776264    2563 scope.go:117] "RemoveContainer" containerID="4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1"
Sep 14 11:29:06.776608 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:29:06.776587    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1\": container with ID starting with 4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1 not found: ID does not exist" containerID="4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1"
Sep 14 11:29:06.776678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.776614    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1"} err="failed to get container status \"4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1\": rpc error: code = NotFound desc = could not find container \"4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1\": container with ID starting with 4b896d49d81ccedf74cfd311421330315d73eed5ca65b46a09eb67064d69fec1 not found: ID does not exist"
Sep 14 11:29:06.776678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.776634    2563 scope.go:117] "RemoveContainer" containerID="1c8895c82e87fc68f69999368831c1ab40bfe31f86367e20e11e93c238efaeea"
Sep 14 11:29:06.781370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.781350    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"]
Sep 14 11:29:06.786571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.786546    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/simulator-w0-6f3d0f-kserve-74fb56dc8-mk8ch"]
Sep 14 11:29:06.786854 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.786834    2563 scope.go:117] "RemoveContainer" containerID="06640bc5ba4f399c5fb0d0a1b0a7c7e9d06cefd19a5ef6bd724469d4790090c1"
Sep 14 11:29:06.799519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.799483    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"]
Sep 14 11:29:06.802755 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.802732    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6f3d0f-7c5449996b-cc77x"]
Sep 14 11:29:06.817745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.817706    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"]
Sep 14 11:29:06.821193 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.821165    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6f3d0f-bf7577949-5fnlw"]
Sep 14 11:29:06.830018 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.829990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"]
Sep 14 11:29:06.833260 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.833238    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6f3d0f/distinct-w0-6f3d0f-kserve-85c9cf9866-d5vwx"]
Sep 14 11:29:06.911440 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:06.911381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:07.705548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:07.705517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:07.919830 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:07.919794    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c8ba12-68b4-4969-ac93-93aff2b49e97" path="/var/lib/kubelet/pods/56c8ba12-68b4-4969-ac93-93aff2b49e97/volumes"
Sep 14 11:29:07.920196 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:07.920171    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722b86d3-0e85-4f0c-b26b-21d07884e2b6" path="/var/lib/kubelet/pods/722b86d3-0e85-4f0c-b26b-21d07884e2b6/volumes"
Sep 14 11:29:07.920553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:07.920539    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b307312c-c91a-4039-a029-40d098e3e3a5" path="/var/lib/kubelet/pods/b307312c-c91a-4039-a029-40d098e3e3a5/volumes"
Sep 14 11:29:07.920910 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:07.920899    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" path="/var/lib/kubelet/pods/f002c3af-b9f8-420a-8a8f-f6b7c05b0e64/volumes"
Sep 14 11:29:08.428086 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:08.428050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:09.377541 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:09.377502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:10.144536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:10.144505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:10.898624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:10.898594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:12.603301 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:12.603257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:12.706952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:12.706914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:12.799490 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:12.799431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:12.898698 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:12.898608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:13.044940 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:13.044893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:13.144514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:13.144485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:13.264277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:13.264190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:13.366924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:13.366892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:14.142219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:14.142139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:14.947144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:14.947111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:15.743288 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:15.743252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:16.506664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:16.506627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:17.420714 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:17.420679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:18.105893 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:18.105852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:19.537911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:19.537874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:19.639855 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:19.639821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:19.964983 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:19.964950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:21.108091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:21.108051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:21.817534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:21.817492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:22.575180 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:22.575146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:23.580361 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:23.580325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:24.420508 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:24.420468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:25.284628 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:25.284597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:26.117875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:26.117832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:26.840201 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:26.840165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:27.619540 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:27.619506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:27.697453 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:27.697422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:28.060900 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:28.060816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:29:28.924681 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:28.924643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:30.390523 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:30.390430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:35.993583 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:35.993547    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"]
Sep 14 11:29:35.994051 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:35.993768    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c" podUID="df8ccdf3-9ea0-42ec-bd84-62d76140ce22" containerName="maas-api" containerID="cri-o://78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0" gracePeriod=30
Sep 14 11:29:36.026703 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.026666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:36.124049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.124013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:36.223148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.223106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:36.239404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.239364    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:29:36.295834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.295761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-metrics-certs\") pod \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") "
Sep 14 11:29:36.295834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.295814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-maas-api-tls\") pod \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") "
Sep 14 11:29:36.296048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.295861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2gjz\" (UniqueName: \"kubernetes.io/projected/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-kube-api-access-f2gjz\") pod \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") "
Sep 14 11:29:36.296048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.295914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-openshift-service-ca\") pod \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\" (UID: \"df8ccdf3-9ea0-42ec-bd84-62d76140ce22\") "
Sep 14 11:29:36.296315 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.296286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df8ccdf3-9ea0-42ec-bd84-62d76140ce22" (UID: "df8ccdf3-9ea0-42ec-bd84-62d76140ce22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:29:36.297797 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.297777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df8ccdf3-9ea0-42ec-bd84-62d76140ce22" (UID: "df8ccdf3-9ea0-42ec-bd84-62d76140ce22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:29:36.297959 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.297939    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df8ccdf3-9ea0-42ec-bd84-62d76140ce22" (UID: "df8ccdf3-9ea0-42ec-bd84-62d76140ce22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:29:36.298021 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.297973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-kube-api-access-f2gjz" (OuterVolumeSpecName: "kube-api-access-f2gjz") pod "df8ccdf3-9ea0-42ec-bd84-62d76140ce22" (UID: "df8ccdf3-9ea0-42ec-bd84-62d76140ce22"). InnerVolumeSpecName "kube-api-access-f2gjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:29:36.331862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.331827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:36.396988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.396945    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gjz\" (UniqueName: \"kubernetes.io/projected/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-kube-api-access-f2gjz\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:36.396988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.396985    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:36.396988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.396994    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:36.396988 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.397003    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8ccdf3-9ea0-42ec-bd84-62d76140ce22-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:29:36.629351 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.629321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:36.779690 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.779651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:36.946020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.945936    2563 generic.go:356] "Generic (PLEG): container finished" podID="df8ccdf3-9ea0-42ec-bd84-62d76140ce22" containerID="78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0" exitCode=0
Sep 14 11:29:36.946020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.945991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c" event={"ID":"df8ccdf3-9ea0-42ec-bd84-62d76140ce22","Type":"ContainerDied","Data":"78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0"}
Sep 14 11:29:36.946207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.946032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c" event={"ID":"df8ccdf3-9ea0-42ec-bd84-62d76140ce22","Type":"ContainerDied","Data":"5d9f1cc5eb830adbcc7f3fb5fc9031192c7ad37273a1bf05dbe88eed1daf55e0"}
Sep 14 11:29:36.946207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.946053    2563 scope.go:117] "RemoveContainer" containerID="78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0"
Sep 14 11:29:36.946207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.945993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"
Sep 14 11:29:36.956901 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.956882    2563 scope.go:117] "RemoveContainer" containerID="78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0"
Sep 14 11:29:36.957172 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:29:36.957152    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0\": container with ID starting with 78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0 not found: ID does not exist" containerID="78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0"
Sep 14 11:29:36.957241 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.957179    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0"} err="failed to get container status \"78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0\": rpc error: code = NotFound desc = could not find container \"78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0\": container with ID starting with 78f16d7daa3aedaa78e12999d70d07a05ecf66a1a041ab4a0e9e67ac808690c0 not found: ID does not exist"
Sep 14 11:29:36.968083 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.968055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"]
Sep 14 11:29:36.971012 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:36.970989    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-420d64-8695bd86b6-cc52c"]
Sep 14 11:29:37.511525 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:37.511487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:37.919514 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:37.919482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df8ccdf3-9ea0-42ec-bd84-62d76140ce22" path="/var/lib/kubelet/pods/df8ccdf3-9ea0-42ec-bd84-62d76140ce22/volumes"
Sep 14 11:29:38.296443 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:38.296335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:39.057110 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:39.057077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:39.798015 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:39.797979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:40.596507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:40.596463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:42.837971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:42.837932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:42.904708 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:42.904676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:43.140899 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:43.140857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:44.431569 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:44.431530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:45.128619 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:45.128578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:46.116277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:46.116238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:46.925572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:46.925535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:47.794269 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:47.794212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:48.512197 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:48.512143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:53.711783 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:53.711746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:54.156807 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:54.156776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:54.232782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:54.232744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:54.306224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:54.306183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:54.381762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:54.381702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:54.734751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:54.734697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:54.855373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:54.855339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:55.602629 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:55.602597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:56.393181 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:56.393145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:57.197451 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:57.197412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:57.937266 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:57.937233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:29:59.261795 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:29:59.261759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:30:00.488214 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:00.488174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:03.142277 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:03.142234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:03.367839 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:03.367803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:04.571937 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:04.571895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:04.687527 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:04.687495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:05.107312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:05.107274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:05.812107 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:05.812076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:06.762377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:06.762341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:08.125834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:08.125785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:08.403869 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:08.403764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:09.120794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:09.120757    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"]
Sep 14 11:30:09.129143 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:09.129112    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"]
Sep 14 11:30:09.141903 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:09.141869    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"]
Sep 14 11:30:09.219447 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:09.219381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:12.276932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.276868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:12.852046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.852012    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"]
Sep 14 11:30:12.852333 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.852307    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6" podUID="1b810f95-2179-462d-b8ea-0c2dbd88f5ab" containerName="maas-api" containerID="cri-o://708c78c3e8f6c7b50ba333ecf38b410f9f0d93223857ef4dcb0bb1d3fc888649" gracePeriod=30
Sep 14 11:30:12.899129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.899097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:12.928231 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.928199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"]
Sep 14 11:30:12.928440 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.928413    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2" podUID="50945937-89ab-44c7-9554-2b844f715778" containerName="payload-pre-processing" containerID="cri-o://b8bde304651ac7ef3251c266f5249d68a697f811f98c2424dabeebb478c76e4a" gracePeriod=30
Sep 14 11:30:12.980247 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:12.980212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:13.080119 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.080081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:13.157317 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.157281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:13.166064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.166027    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b810f95-2179-462d-b8ea-0c2dbd88f5ab" containerID="708c78c3e8f6c7b50ba333ecf38b410f9f0d93223857ef4dcb0bb1d3fc888649" exitCode=0
Sep 14 11:30:13.166151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.166098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6" event={"ID":"1b810f95-2179-462d-b8ea-0c2dbd88f5ab","Type":"ContainerDied","Data":"708c78c3e8f6c7b50ba333ecf38b410f9f0d93223857ef4dcb0bb1d3fc888649"}
Sep 14 11:30:13.168048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.168026    2563 generic.go:356] "Generic (PLEG): container finished" podID="50945937-89ab-44c7-9554-2b844f715778" containerID="b8bde304651ac7ef3251c266f5249d68a697f811f98c2424dabeebb478c76e4a" exitCode=0
Sep 14 11:30:13.168162 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.168099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2" event={"ID":"50945937-89ab-44c7-9554-2b844f715778","Type":"ContainerDied","Data":"b8bde304651ac7ef3251c266f5249d68a697f811f98c2424dabeebb478c76e4a"}
Sep 14 11:30:13.182992 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.182972    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:30:13.209265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.209243    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:30:13.341610 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.341580    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50945937-89ab-44c7-9554-2b844f715778-plugins-config-volume\") pod \"50945937-89ab-44c7-9554-2b844f715778\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") "
Sep 14 11:30:13.342029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.341632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-openshift-service-ca\") pod \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") "
Sep 14 11:30:13.342029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.341697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-metrics-certs\") pod \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") "
Sep 14 11:30:13.342029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.341725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnxx9\" (UniqueName: \"kubernetes.io/projected/50945937-89ab-44c7-9554-2b844f715778-kube-api-access-bnxx9\") pod \"50945937-89ab-44c7-9554-2b844f715778\" (UID: \"50945937-89ab-44c7-9554-2b844f715778\") "
Sep 14 11:30:13.342029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.341745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-maas-api-tls\") pod \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") "
Sep 14 11:30:13.342029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.341766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjkxz\" (UniqueName: \"kubernetes.io/projected/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-kube-api-access-xjkxz\") pod \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\" (UID: \"1b810f95-2179-462d-b8ea-0c2dbd88f5ab\") "
Sep 14 11:30:13.342029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.342019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50945937-89ab-44c7-9554-2b844f715778-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50945937-89ab-44c7-9554-2b844f715778" (UID: "50945937-89ab-44c7-9554-2b844f715778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:30:13.342285 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.342030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b810f95-2179-462d-b8ea-0c2dbd88f5ab" (UID: "1b810f95-2179-462d-b8ea-0c2dbd88f5ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:30:13.343948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.343928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b810f95-2179-462d-b8ea-0c2dbd88f5ab" (UID: "1b810f95-2179-462d-b8ea-0c2dbd88f5ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:13.344020 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.343930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-kube-api-access-xjkxz" (OuterVolumeSpecName: "kube-api-access-xjkxz") pod "1b810f95-2179-462d-b8ea-0c2dbd88f5ab" (UID: "1b810f95-2179-462d-b8ea-0c2dbd88f5ab"). InnerVolumeSpecName "kube-api-access-xjkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:30:13.344238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.344213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b810f95-2179-462d-b8ea-0c2dbd88f5ab" (UID: "1b810f95-2179-462d-b8ea-0c2dbd88f5ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:13.344340 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.344221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50945937-89ab-44c7-9554-2b844f715778-kube-api-access-bnxx9" (OuterVolumeSpecName: "kube-api-access-bnxx9") pod "50945937-89ab-44c7-9554-2b844f715778" (UID: "50945937-89ab-44c7-9554-2b844f715778"). InnerVolumeSpecName "kube-api-access-bnxx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:30:13.442744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.442679    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:13.442744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.442702    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxx9\" (UniqueName: \"kubernetes.io/projected/50945937-89ab-44c7-9554-2b844f715778-kube-api-access-bnxx9\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:13.442744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.442712    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:13.442744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.442721    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkxz\" (UniqueName: \"kubernetes.io/projected/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-kube-api-access-xjkxz\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:13.442744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.442730    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50945937-89ab-44c7-9554-2b844f715778-plugins-config-volume\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:13.442744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.442738    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b810f95-2179-462d-b8ea-0c2dbd88f5ab-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:13.474691 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.474659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:13.557810 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:13.557777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:14.175527 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.175451    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"
Sep 14 11:30:14.175672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.175456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6" event={"ID":"1b810f95-2179-462d-b8ea-0c2dbd88f5ab","Type":"ContainerDied","Data":"34cb1ae7ade6d350af8cbc1b2efcd47ab870eed0ca82ad8e570c4365b6557251"}
Sep 14 11:30:14.175672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.175603    2563 scope.go:117] "RemoveContainer" containerID="708c78c3e8f6c7b50ba333ecf38b410f9f0d93223857ef4dcb0bb1d3fc888649"
Sep 14 11:30:14.177091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.176910    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"
Sep 14 11:30:14.177091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.176962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2" event={"ID":"50945937-89ab-44c7-9554-2b844f715778","Type":"ContainerDied","Data":"f8b614c9f702e14aaa6c57a89b19ed13b4da71c87de3f180357c0b2caf32de90"}
Sep 14 11:30:14.185587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.185567    2563 scope.go:117] "RemoveContainer" containerID="b8bde304651ac7ef3251c266f5249d68a697f811f98c2424dabeebb478c76e4a"
Sep 14 11:30:14.194682 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.194632    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"]
Sep 14 11:30:14.197911 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.197890    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-41ae25-5856596bd9-6v8h6"]
Sep 14 11:30:14.207052 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.207029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"]
Sep 14 11:30:14.209230 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.209214    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-41ae25-6d8f7cc87d-5zrp2"]
Sep 14 11:30:14.579763 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:14.579679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:15.294996 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:15.294960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:15.918746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:15.918714    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b810f95-2179-462d-b8ea-0c2dbd88f5ab" path="/var/lib/kubelet/pods/1b810f95-2179-462d-b8ea-0c2dbd88f5ab/volumes"
Sep 14 11:30:15.919136 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:15.919094    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50945937-89ab-44c7-9554-2b844f715778" path="/var/lib/kubelet/pods/50945937-89ab-44c7-9554-2b844f715778/volumes"
Sep 14 11:30:19.553775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:19.553734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:19.653740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:19.653694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:19.823754 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:19.823653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:20.993882 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:20.993848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:21.084092 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:21.084050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:21.183405 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:21.183364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:21.282933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:21.282844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:21.619516 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:21.619486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:21.721764 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:21.721723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:22.147965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:22.147928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:30:22.529323 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:22.529232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:23.293042 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:23.293000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:24.141063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.141018    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z" podUID="c5a4e4ef-93df-43e5-bd63-0c02fb987c26" containerName="main" containerID="cri-o://781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a" gracePeriod=45
Sep 14 11:30:24.150854 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.150815    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg" podUID="8e086787-53a2-4d1b-b0fa-e0d167cc7138" containerName="main" containerID="cri-o://b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e" gracePeriod=45
Sep 14 11:30:24.165793 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.165761    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz" podUID="b3f2b565-08d1-4a1c-84de-b1db06ca174c" containerName="main" containerID="cri-o://8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6" gracePeriod=45
Sep 14 11:30:24.466612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.466587    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:30:24.469639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.469616    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:30:24.473436 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.473415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:30:24.551959 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.551926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tmp-dir\") pod \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") "
Sep 14 11:30:24.552146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.551972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-dshm\") pod \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") "
Sep 14 11:30:24.552146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-home\") pod \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") "
Sep 14 11:30:24.552146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-dshm\") pod \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") "
Sep 14 11:30:24.552146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tmp-dir\") pod \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") "
Sep 14 11:30:24.552146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552109    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cnmm\" (UniqueName: \"kubernetes.io/projected/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-kube-api-access-5cnmm\") pod \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") "
Sep 14 11:30:24.552146 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tls-certs\") pod \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-home\") pod \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tls-certs\") pod \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552213    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-model-cache\") pod \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552240    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6n78\" (UniqueName: \"kubernetes.io/projected/b3f2b565-08d1-4a1c-84de-b1db06ca174c-kube-api-access-n6n78\") pod \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9vsj\" (UniqueName: \"kubernetes.io/projected/8e086787-53a2-4d1b-b0fa-e0d167cc7138-kube-api-access-r9vsj\") pod \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-dshm\") pod \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") "
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3f2b565-08d1-4a1c-84de-b1db06ca174c" (UID: "b3f2b565-08d1-4a1c-84de-b1db06ca174c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-home" (OuterVolumeSpecName: "home") pod "c5a4e4ef-93df-43e5-bd63-0c02fb987c26" (UID: "c5a4e4ef-93df-43e5-bd63-0c02fb987c26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.552496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.552480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-home" (OuterVolumeSpecName: "home") pod "b3f2b565-08d1-4a1c-84de-b1db06ca174c" (UID: "b3f2b565-08d1-4a1c-84de-b1db06ca174c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-model-cache\") pod \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") "
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554451    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-model-cache\") pod \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") "
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tls-certs\") pod \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\" (UID: \"b3f2b565-08d1-4a1c-84de-b1db06ca174c\") "
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-home\") pod \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\" (UID: \"8e086787-53a2-4d1b-b0fa-e0d167cc7138\") "
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tmp-dir\") pod \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\" (UID: \"c5a4e4ef-93df-43e5-bd63-0c02fb987c26\") "
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e086787-53a2-4d1b-b0fa-e0d167cc7138" (UID: "8e086787-53a2-4d1b-b0fa-e0d167cc7138"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.554994    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555034    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555053    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555064    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e086787-53a2-4d1b-b0fa-e0d167cc7138" (UID: "8e086787-53a2-4d1b-b0fa-e0d167cc7138"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5a4e4ef-93df-43e5-bd63-0c02fb987c26" (UID: "c5a4e4ef-93df-43e5-bd63-0c02fb987c26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555247    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3f2b565-08d1-4a1c-84de-b1db06ca174c" (UID: "b3f2b565-08d1-4a1c-84de-b1db06ca174c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e086787-53a2-4d1b-b0fa-e0d167cc7138-kube-api-access-r9vsj" (OuterVolumeSpecName: "kube-api-access-r9vsj") pod "8e086787-53a2-4d1b-b0fa-e0d167cc7138" (UID: "8e086787-53a2-4d1b-b0fa-e0d167cc7138"). InnerVolumeSpecName "kube-api-access-r9vsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-dshm" (OuterVolumeSpecName: "dshm") pod "b3f2b565-08d1-4a1c-84de-b1db06ca174c" (UID: "b3f2b565-08d1-4a1c-84de-b1db06ca174c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-home" (OuterVolumeSpecName: "home") pod "8e086787-53a2-4d1b-b0fa-e0d167cc7138" (UID: "8e086787-53a2-4d1b-b0fa-e0d167cc7138"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.555571 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5a4e4ef-93df-43e5-bd63-0c02fb987c26" (UID: "c5a4e4ef-93df-43e5-bd63-0c02fb987c26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.556615 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.555754    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e086787-53a2-4d1b-b0fa-e0d167cc7138" (UID: "8e086787-53a2-4d1b-b0fa-e0d167cc7138"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:24.556867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.556842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f2b565-08d1-4a1c-84de-b1db06ca174c-kube-api-access-n6n78" (OuterVolumeSpecName: "kube-api-access-n6n78") pod "b3f2b565-08d1-4a1c-84de-b1db06ca174c" (UID: "b3f2b565-08d1-4a1c-84de-b1db06ca174c"). InnerVolumeSpecName "kube-api-access-n6n78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:30:24.556867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.556856    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-dshm" (OuterVolumeSpecName: "dshm") pod "8e086787-53a2-4d1b-b0fa-e0d167cc7138" (UID: "8e086787-53a2-4d1b-b0fa-e0d167cc7138"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.557220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.557201    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5a4e4ef-93df-43e5-bd63-0c02fb987c26" (UID: "c5a4e4ef-93df-43e5-bd63-0c02fb987c26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:24.557426 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.557372    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3f2b565-08d1-4a1c-84de-b1db06ca174c" (UID: "b3f2b565-08d1-4a1c-84de-b1db06ca174c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:30:24.558548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.558530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-kube-api-access-5cnmm" (OuterVolumeSpecName: "kube-api-access-5cnmm") pod "c5a4e4ef-93df-43e5-bd63-0c02fb987c26" (UID: "c5a4e4ef-93df-43e5-bd63-0c02fb987c26"). InnerVolumeSpecName "kube-api-access-5cnmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:30:24.558730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.558701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-dshm" (OuterVolumeSpecName: "dshm") pod "c5a4e4ef-93df-43e5-bd63-0c02fb987c26" (UID: "c5a4e4ef-93df-43e5-bd63-0c02fb987c26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:30:24.656408 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656361    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656427    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f2b565-08d1-4a1c-84de-b1db06ca174c-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656437    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656446    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656453    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656460    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656467    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656476    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnmm\" (UniqueName: \"kubernetes.io/projected/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-kube-api-access-5cnmm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656485    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e086787-53a2-4d1b-b0fa-e0d167cc7138-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656493    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656500    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b565-08d1-4a1c-84de-b1db06ca174c-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656508    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6n78\" (UniqueName: \"kubernetes.io/projected/b3f2b565-08d1-4a1c-84de-b1db06ca174c-kube-api-access-n6n78\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656516    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vsj\" (UniqueName: \"kubernetes.io/projected/8e086787-53a2-4d1b-b0fa-e0d167cc7138-kube-api-access-r9vsj\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:24.656581 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:24.656523    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5a4e4ef-93df-43e5-bd63-0c02fb987c26-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:30:25.243010 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.242970    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5a4e4ef-93df-43e5-bd63-0c02fb987c26" containerID="781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a" exitCode=0
Sep 14 11:30:25.243194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.243017    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"
Sep 14 11:30:25.243194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.243052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z" event={"ID":"c5a4e4ef-93df-43e5-bd63-0c02fb987c26","Type":"ContainerDied","Data":"781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a"}
Sep 14 11:30:25.243194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.243095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z" event={"ID":"c5a4e4ef-93df-43e5-bd63-0c02fb987c26","Type":"ContainerDied","Data":"097fe895c352546c4e4b081da3a550aef867c866f69c9af5e164f2ae5ad22b3d"}
Sep 14 11:30:25.243194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.243116    2563 scope.go:117] "RemoveContainer" containerID="781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a"
Sep 14 11:30:25.244535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.244443    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3f2b565-08d1-4a1c-84de-b1db06ca174c" containerID="8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6" exitCode=0
Sep 14 11:30:25.244535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.244500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"
Sep 14 11:30:25.244535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.244525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz" event={"ID":"b3f2b565-08d1-4a1c-84de-b1db06ca174c","Type":"ContainerDied","Data":"8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6"}
Sep 14 11:30:25.244711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.244576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz" event={"ID":"b3f2b565-08d1-4a1c-84de-b1db06ca174c","Type":"ContainerDied","Data":"5e70214ecfb1baa8193b4b724140d6bfa4246c50b2af6b4f261333409ad2a55d"}
Sep 14 11:30:25.245926 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.245906    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e086787-53a2-4d1b-b0fa-e0d167cc7138" containerID="b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e" exitCode=0
Sep 14 11:30:25.246033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.245938    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"
Sep 14 11:30:25.246033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.245947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg" event={"ID":"8e086787-53a2-4d1b-b0fa-e0d167cc7138","Type":"ContainerDied","Data":"b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e"}
Sep 14 11:30:25.246033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.245974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg" event={"ID":"8e086787-53a2-4d1b-b0fa-e0d167cc7138","Type":"ContainerDied","Data":"9cffd8faf0a72626790c55e4523f7b9144f83950be3ec9c727079d5af5218e6f"}
Sep 14 11:30:25.253144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.253116    2563 scope.go:117] "RemoveContainer" containerID="781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a"
Sep 14 11:30:25.253457 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:30:25.253432    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a\": container with ID starting with 781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a not found: ID does not exist" containerID="781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a"
Sep 14 11:30:25.253553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.253467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a"} err="failed to get container status \"781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a\": rpc error: code = NotFound desc = could not find container \"781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a\": container with ID starting with 781035c6df3243c2ec654c8ec9e7b26261b809ba5febb54ca20e4b252820f33a not found: ID does not exist"
Sep 14 11:30:25.253553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.253490    2563 scope.go:117] "RemoveContainer" containerID="8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6"
Sep 14 11:30:25.262585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.262569    2563 scope.go:117] "RemoveContainer" containerID="8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6"
Sep 14 11:30:25.262844 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:30:25.262824    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6\": container with ID starting with 8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6 not found: ID does not exist" containerID="8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6"
Sep 14 11:30:25.262920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.262854    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6"} err="failed to get container status \"8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6\": rpc error: code = NotFound desc = could not find container \"8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6\": container with ID starting with 8612d7e1a711a114b9af51a3698183523a41d774ba6533ae63e720a1308fb5c6 not found: ID does not exist"
Sep 14 11:30:25.262920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.262875    2563 scope.go:117] "RemoveContainer" containerID="b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e"
Sep 14 11:30:25.271654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.271635    2563 scope.go:117] "RemoveContainer" containerID="b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e"
Sep 14 11:30:25.271938 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:30:25.271915    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e\": container with ID starting with b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e not found: ID does not exist" containerID="b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e"
Sep 14 11:30:25.272002 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.271943    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e"} err="failed to get container status \"b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e\": rpc error: code = NotFound desc = could not find container \"b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e\": container with ID starting with b1f0f5a83f36f418692b36b6a5d9c8b4a1622d717c1ee1584953ad1802f6e98e not found: ID does not exist"
Sep 14 11:30:25.272953 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.272932    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"]
Sep 14 11:30:25.276482 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.276462    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/distinct-w4-de0f7a-kserve-756f6bfd5d-kmc6z"]
Sep 14 11:30:25.286167 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.286137    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"]
Sep 14 11:30:25.290342 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.290322    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/premium-w4-de0f7a-kserve-564fbf8647-gcgpg"]
Sep 14 11:30:25.300759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.300737    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"]
Sep 14 11:30:25.304522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.304503    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-de0f7a/simulator-w4-de0f7a-kserve-6bdd67c4f8-gfccz"]
Sep 14 11:30:25.919027 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.918983    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e086787-53a2-4d1b-b0fa-e0d167cc7138" path="/var/lib/kubelet/pods/8e086787-53a2-4d1b-b0fa-e0d167cc7138/volumes"
Sep 14 11:30:25.919604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.919583    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f2b565-08d1-4a1c-84de-b1db06ca174c" path="/var/lib/kubelet/pods/b3f2b565-08d1-4a1c-84de-b1db06ca174c/volumes"
Sep 14 11:30:25.920003 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:25.919985    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a4e4ef-93df-43e5-bd63-0c02fb987c26" path="/var/lib/kubelet/pods/c5a4e4ef-93df-43e5-bd63-0c02fb987c26/volumes"
Sep 14 11:30:28.051772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:28.051735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:28.114008 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:28.113970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:28.320656 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:28.320558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:40.848353 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:40.848315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:30:45.959474 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:45.959366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:46.026148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:46.026112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:46.892110 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:46.892072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:52.021073 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:52.021043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:30:52.035247 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:52.035219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:30:52.593397 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:52.593363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:54.981215 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:54.981178    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"]
Sep 14 11:30:55.005987 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:55.005952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:56.045695 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:56.045663    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"]
Sep 14 11:30:56.067517 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:56.067486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:57.055270 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:57.055229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:30:57.238184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:30:57.238146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:31:06.547543 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:06.547509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:31:08.657885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:08.657844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:31:10.000426 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.000370    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s" podUID="2d42808c-99de-4d62-9e31-51bb9f29ff87" containerName="main" containerID="cri-o://5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1" gracePeriod=45
Sep 14 11:31:10.249465 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.249439    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:31:10.260238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-home\") pod \"2d42808c-99de-4d62-9e31-51bb9f29ff87\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") "
Sep 14 11:31:10.260238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d42808c-99de-4d62-9e31-51bb9f29ff87-tls-certs\") pod \"2d42808c-99de-4d62-9e31-51bb9f29ff87\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") "
Sep 14 11:31:10.260468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-dshm\") pod \"2d42808c-99de-4d62-9e31-51bb9f29ff87\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") "
Sep 14 11:31:10.260468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-model-cache\") pod \"2d42808c-99de-4d62-9e31-51bb9f29ff87\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") "
Sep 14 11:31:10.260468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-tmp-dir\") pod \"2d42808c-99de-4d62-9e31-51bb9f29ff87\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") "
Sep 14 11:31:10.260468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flgtt\" (UniqueName: \"kubernetes.io/projected/2d42808c-99de-4d62-9e31-51bb9f29ff87-kube-api-access-flgtt\") pod \"2d42808c-99de-4d62-9e31-51bb9f29ff87\" (UID: \"2d42808c-99de-4d62-9e31-51bb9f29ff87\") "
Sep 14 11:31:10.260468 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-home" (OuterVolumeSpecName: "home") pod "2d42808c-99de-4d62-9e31-51bb9f29ff87" (UID: "2d42808c-99de-4d62-9e31-51bb9f29ff87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:10.260729 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d42808c-99de-4d62-9e31-51bb9f29ff87" (UID: "2d42808c-99de-4d62-9e31-51bb9f29ff87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:10.260729 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260627    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d42808c-99de-4d62-9e31-51bb9f29ff87" (UID: "2d42808c-99de-4d62-9e31-51bb9f29ff87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:10.260803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260731    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:10.260803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260748    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:10.260803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.260762    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:10.262644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.262621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-dshm" (OuterVolumeSpecName: "dshm") pod "2d42808c-99de-4d62-9e31-51bb9f29ff87" (UID: "2d42808c-99de-4d62-9e31-51bb9f29ff87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:10.262742 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.262658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d42808c-99de-4d62-9e31-51bb9f29ff87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d42808c-99de-4d62-9e31-51bb9f29ff87" (UID: "2d42808c-99de-4d62-9e31-51bb9f29ff87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:31:10.262742 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.262684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d42808c-99de-4d62-9e31-51bb9f29ff87-kube-api-access-flgtt" (OuterVolumeSpecName: "kube-api-access-flgtt") pod "2d42808c-99de-4d62-9e31-51bb9f29ff87" (UID: "2d42808c-99de-4d62-9e31-51bb9f29ff87"). InnerVolumeSpecName "kube-api-access-flgtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:31:10.361376 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.361339    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d42808c-99de-4d62-9e31-51bb9f29ff87-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:10.361376 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.361369    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d42808c-99de-4d62-9e31-51bb9f29ff87-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:10.361376 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.361378    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flgtt\" (UniqueName: \"kubernetes.io/projected/2d42808c-99de-4d62-9e31-51bb9f29ff87-kube-api-access-flgtt\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:10.509046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.509006    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d42808c-99de-4d62-9e31-51bb9f29ff87" containerID="5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1" exitCode=0
Sep 14 11:31:10.509046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.509049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"
Sep 14 11:31:10.509321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.509095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s" event={"ID":"2d42808c-99de-4d62-9e31-51bb9f29ff87","Type":"ContainerDied","Data":"5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1"}
Sep 14 11:31:10.509321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.509150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s" event={"ID":"2d42808c-99de-4d62-9e31-51bb9f29ff87","Type":"ContainerDied","Data":"1c68c4857faa486211f117018b46b28d1804e1e0899d2597b9650c1e4077d411"}
Sep 14 11:31:10.509321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.509173    2563 scope.go:117] "RemoveContainer" containerID="5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1"
Sep 14 11:31:10.518896 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.518877    2563 scope.go:117] "RemoveContainer" containerID="5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1"
Sep 14 11:31:10.519204 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:31:10.519185    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1\": container with ID starting with 5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1 not found: ID does not exist" containerID="5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1"
Sep 14 11:31:10.519250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.519214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1"} err="failed to get container status \"5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1\": rpc error: code = NotFound desc = could not find container \"5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1\": container with ID starting with 5b9870b7807caacbf1ecbfb91e2568b6187d0c89870c1fb28cec1366639a28c1 not found: ID does not exist"
Sep 14 11:31:10.530074 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.530044    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"]
Sep 14 11:31:10.535913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:10.535889    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/auth-test-model-55c025-kserve-54c6679db7-5qc9s"]
Sep 14 11:31:11.064683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.064624    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p" podUID="53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" containerName="main" containerID="cri-o://f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3" gracePeriod=45
Sep 14 11:31:11.320108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.320051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:31:11.367659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-model-cache\") pod \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") "
Sep 14 11:31:11.367659 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tls-certs\") pod \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") "
Sep 14 11:31:11.367908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-home\") pod \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") "
Sep 14 11:31:11.367908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tmp-dir\") pod \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") "
Sep 14 11:31:11.367908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-297h7\" (UniqueName: \"kubernetes.io/projected/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-kube-api-access-297h7\") pod \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") "
Sep 14 11:31:11.367908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367794    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-dshm\") pod \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\" (UID: \"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4\") "
Sep 14 11:31:11.368133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-home" (OuterVolumeSpecName: "home") pod "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" (UID: "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:11.368133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.367941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" (UID: "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:11.368133 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.368018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" (UID: "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:11.368264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.368144    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:11.368264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.368160    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:11.368264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.368173    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:11.370284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.370261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" (UID: "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:31:11.370446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.370374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-dshm" (OuterVolumeSpecName: "dshm") pod "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" (UID: "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:31:11.370446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.370410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-kube-api-access-297h7" (OuterVolumeSpecName: "kube-api-access-297h7") pod "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" (UID: "53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4"). InnerVolumeSpecName "kube-api-access-297h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:31:11.469370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.469331    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:11.469370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.469364    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-297h7\" (UniqueName: \"kubernetes.io/projected/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-kube-api-access-297h7\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:11.469370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.469374    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:31:11.517404 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.517358    2563 generic.go:356] "Generic (PLEG): container finished" podID="53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" containerID="f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3" exitCode=0
Sep 14 11:31:11.517574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.517421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"
Sep 14 11:31:11.517574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.517432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p" event={"ID":"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4","Type":"ContainerDied","Data":"f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3"}
Sep 14 11:31:11.517574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.517466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p" event={"ID":"53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4","Type":"ContainerDied","Data":"afcc105932f6ee3042d49925da8edead9a97456e29dc942bfc31ec4d8407a965"}
Sep 14 11:31:11.517574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.517484    2563 scope.go:117] "RemoveContainer" containerID="f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3"
Sep 14 11:31:11.526927 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.526908    2563 scope.go:117] "RemoveContainer" containerID="f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3"
Sep 14 11:31:11.527188 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:31:11.527169    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3\": container with ID starting with f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3 not found: ID does not exist" containerID="f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3"
Sep 14 11:31:11.527268 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.527195    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3"} err="failed to get container status \"f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3\": rpc error: code = NotFound desc = could not find container \"f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3\": container with ID starting with f8e52174f210d89972525cf73c29095d42f2f6aaadc47ff7715b963b8f2c33c3 not found: ID does not exist"
Sep 14 11:31:11.538938 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.538912    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"]
Sep 14 11:31:11.542955 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.542933    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/auth-test-model-3f02a5-kserve-6bc497fd55-zwt6p"]
Sep 14 11:31:11.918945 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.918916    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d42808c-99de-4d62-9e31-51bb9f29ff87" path="/var/lib/kubelet/pods/2d42808c-99de-4d62-9e31-51bb9f29ff87/volumes"
Sep 14 11:31:11.919346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:11.919333    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" path="/var/lib/kubelet/pods/53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4/volumes"
Sep 14 11:31:16.750236 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:16.750197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:31:35.004365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:35.004326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:31:43.047812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.047772    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"]
Sep 14 11:31:43.048460 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df8ccdf3-9ea0-42ec-bd84-62d76140ce22" containerName="maas-api"
Sep 14 11:31:43.048518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df8ccdf3-9ea0-42ec-bd84-62d76140ce22" containerName="maas-api"
Sep 14 11:31:43.048518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048485    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" containerName="main"
Sep 14 11:31:43.048518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048493    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" containerName="main"
Sep 14 11:31:43.048518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048517    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d42808c-99de-4d62-9e31-51bb9f29ff87" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d42808c-99de-4d62-9e31-51bb9f29ff87" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048535    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b307312c-c91a-4039-a029-40d098e3e3a5" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b307312c-c91a-4039-a029-40d098e3e3a5" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a4e4ef-93df-43e5-bd63-0c02fb987c26" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048564    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a4e4ef-93df-43e5-bd63-0c02fb987c26" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b810f95-2179-462d-b8ea-0c2dbd88f5ab" containerName="maas-api"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048583    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b810f95-2179-462d-b8ea-0c2dbd88f5ab" containerName="maas-api"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048596    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722b86d3-0e85-4f0c-b26b-21d07884e2b6" containerName="payload-processing"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048605    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="722b86d3-0e85-4f0c-b26b-21d07884e2b6" containerName="payload-processing"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e086787-53a2-4d1b-b0fa-e0d167cc7138" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048622    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e086787-53a2-4d1b-b0fa-e0d167cc7138" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" containerName="main"
Sep 14 11:31:43.048647 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048650    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048667    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50945937-89ab-44c7-9554-2b844f715778" containerName="payload-pre-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50945937-89ab-44c7-9554-2b844f715778" containerName="payload-pre-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048695    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c8ba12-68b4-4969-ac93-93aff2b49e97" containerName="payload-pre-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048704    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c8ba12-68b4-4969-ac93-93aff2b49e97" containerName="payload-pre-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048715    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f2b565-08d1-4a1c-84de-b1db06ca174c" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048723    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f2b565-08d1-4a1c-84de-b1db06ca174c" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048822    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d42808c-99de-4d62-9e31-51bb9f29ff87" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048836    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53d1a20e-f89c-4b3e-9f94-8d51c59eeeb4" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048846    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c8ba12-68b4-4969-ac93-93aff2b49e97" containerName="payload-pre-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048857    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e086787-53a2-4d1b-b0fa-e0d167cc7138" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f2b565-08d1-4a1c-84de-b1db06ca174c" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048875    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b810f95-2179-462d-b8ea-0c2dbd88f5ab" containerName="maas-api"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048882    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a4e4ef-93df-43e5-bd63-0c02fb987c26" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048891    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50945937-89ab-44c7-9554-2b844f715778" containerName="payload-pre-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048900    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f002c3af-b9f8-420a-8a8f-f6b7c05b0e64" containerName="main"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048908    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df8ccdf3-9ea0-42ec-bd84-62d76140ce22" containerName="maas-api"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048919    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="722b86d3-0e85-4f0c-b26b-21d07884e2b6" containerName="payload-processing"
Sep 14 11:31:43.049078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.048926    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b307312c-c91a-4039-a029-40d098e3e3a5" containerName="main"
Sep 14 11:31:43.052216 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.052197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.054452 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.054431    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"kube-root-ca.crt\""
Sep 14 11:31:43.055050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.055029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"default-dockercfg-q828q\""
Sep 14 11:31:43.055160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.055049    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"openshift-service-ca.crt\""
Sep 14 11:31:43.055160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.055049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"sub-test-model-3f02a5-kserve-self-signed-certs\""
Sep 14 11:31:43.061861 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.061834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"]
Sep 14 11:31:43.082711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.082682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:31:43.152683 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.152649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-model-cache\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.152828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.152690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-home\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.152828 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.152800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-tmp-dir\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.152920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.152833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjh4p\" (UniqueName: \"kubernetes.io/projected/a022f7fb-6e2b-4bcb-9807-194915530dea-kube-api-access-sjh4p\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.152920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.152907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a022f7fb-6e2b-4bcb-9807-194915530dea-tls-certs\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.152993 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.152931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-dshm\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-model-cache\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-home\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-tmp-dir\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjh4p\" (UniqueName: \"kubernetes.io/projected/a022f7fb-6e2b-4bcb-9807-194915530dea-kube-api-access-sjh4p\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a022f7fb-6e2b-4bcb-9807-194915530dea-tls-certs\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254368 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-dshm\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-model-cache\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254652 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-tmp-dir\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.254652 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.254633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-home\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.256614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.256592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-dshm\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.256764 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.256747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a022f7fb-6e2b-4bcb-9807-194915530dea-tls-certs\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.267151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.267128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjh4p\" (UniqueName: \"kubernetes.io/projected/a022f7fb-6e2b-4bcb-9807-194915530dea-kube-api-access-sjh4p\") pod \"sub-test-model-3f02a5-kserve-744ddf854f-p96cb\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.363072 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.363044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:43.635120 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.635091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"]
Sep 14 11:31:43.636910 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:31:43.636885    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda022f7fb_6e2b_4bcb_9807_194915530dea.slice/crio-f5d4e5038a55268ac8587dcb5a62ab81ca94ba1f8ad0839f930a7ff8162aea09 WatchSource:0}: Error finding container f5d4e5038a55268ac8587dcb5a62ab81ca94ba1f8ad0839f930a7ff8162aea09: Status 404 returned error can't find the container with id f5d4e5038a55268ac8587dcb5a62ab81ca94ba1f8ad0839f930a7ff8162aea09
Sep 14 11:31:43.701878 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.701840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb" event={"ID":"a022f7fb-6e2b-4bcb-9807-194915530dea","Type":"ContainerStarted","Data":"0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542"}
Sep 14 11:31:43.702006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.701891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb" event={"ID":"a022f7fb-6e2b-4bcb-9807-194915530dea","Type":"ContainerStarted","Data":"f5d4e5038a55268ac8587dcb5a62ab81ca94ba1f8ad0839f930a7ff8162aea09"}
Sep 14 11:31:43.721685 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:43.721637    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb" podStartSLOduration=0.721617541 podStartE2EDuration="721.617541ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:31:43.718213319 +0000 UTC m=+1852.340766546" watchObservedRunningTime="2026-09-14 11:31:43.721617541 +0000 UTC m=+1852.344170795"
Sep 14 11:31:46.646191 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:46.646146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:31:53.363995 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:53.363957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:53.364370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:53.364204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:53.376503 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:53.376471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:53.769049 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:53.768953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:31:54.424650 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:54.424614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:31:54.484913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:31:54.484887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:09.145985 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:09.145935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:32:19.582780 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.582690    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-3f4451"]
Sep 14 11:32:19.586435 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.586416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:19.588377 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.588358    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5kv55\""
Sep 14 11:32:19.595410 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.595365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-3f4451"]
Sep 14 11:32:19.684043 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.684015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7m2q\" (UniqueName: \"kubernetes.io/projected/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9-kube-api-access-x7m2q\") pod \"test-curl-7938-3f4451\" (UID: \"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9\") " pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:19.768644 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.768604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:19.784730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.784698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7m2q\" (UniqueName: \"kubernetes.io/projected/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9-kube-api-access-x7m2q\") pod \"test-curl-7938-3f4451\" (UID: \"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9\") " pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:19.792805 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.792774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7m2q\" (UniqueName: \"kubernetes.io/projected/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9-kube-api-access-x7m2q\") pod \"test-curl-7938-3f4451\" (UID: \"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9\") " pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:19.896088 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:19.896049    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:20.166542 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:20.166516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-3f4451"]
Sep 14 11:32:20.168731 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:32:20.168693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb3ed41_1f83_44bb_bf2f_cfaa120d41b9.slice/crio-4e92a70ca9b33fd503d1da9f35bcb2b5d010ce04f1a5412827be29c018b2e9c3 WatchSource:0}: Error finding container 4e92a70ca9b33fd503d1da9f35bcb2b5d010ce04f1a5412827be29c018b2e9c3: Status 404 returned error can't find the container with id 4e92a70ca9b33fd503d1da9f35bcb2b5d010ce04f1a5412827be29c018b2e9c3
Sep 14 11:32:20.910118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:20.910084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-3f4451" event={"ID":"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9","Type":"ContainerStarted","Data":"e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f"}
Sep 14 11:32:20.910118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:20.910125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-3f4451" event={"ID":"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9","Type":"ContainerStarted","Data":"4e92a70ca9b33fd503d1da9f35bcb2b5d010ce04f1a5412827be29c018b2e9c3"}
Sep 14 11:32:20.925887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:20.925841    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-3f4451" podStartSLOduration=1.925826386 podStartE2EDuration="1.925826386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:32:20.925132088 +0000 UTC m=+1889.547685302" watchObservedRunningTime="2026-09-14 11:32:20.925826386 +0000 UTC m=+1889.548379600"
Sep 14 11:32:21.511511 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.511469    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-3f4451"]
Sep 14 11:32:21.515335 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.515305    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-3f4451"]
Sep 14 11:32:21.634565 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.634532    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-729f9e"]
Sep 14 11:32:21.634979 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.634965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" containerName="test-curl-7938-3f4451"
Sep 14 11:32:21.635032 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.634981    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" containerName="test-curl-7938-3f4451"
Sep 14 11:32:21.635071 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.635046    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" containerName="test-curl-7938-3f4451"
Sep 14 11:32:21.638238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.638221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:21.645304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.645279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-729f9e"]
Sep 14 11:32:21.701937 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.701904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq8c\" (UniqueName: \"kubernetes.io/projected/be608ddd-4bde-4706-b941-2ea209c0db52-kube-api-access-qqq8c\") pod \"test-curl-7938-729f9e\" (UID: \"be608ddd-4bde-4706-b941-2ea209c0db52\") " pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:21.802725 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.802687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq8c\" (UniqueName: \"kubernetes.io/projected/be608ddd-4bde-4706-b941-2ea209c0db52-kube-api-access-qqq8c\") pod \"test-curl-7938-729f9e\" (UID: \"be608ddd-4bde-4706-b941-2ea209c0db52\") " pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:21.810592 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.810563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq8c\" (UniqueName: \"kubernetes.io/projected/be608ddd-4bde-4706-b941-2ea209c0db52-kube-api-access-qqq8c\") pod \"test-curl-7938-729f9e\" (UID: \"be608ddd-4bde-4706-b941-2ea209c0db52\") " pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:21.918320 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:32:21.918228    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-3f4451" container="test-curl-7938-3f4451"
Sep 14 11:32:21.948760 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:21.948734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:22.224951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:22.224925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-729f9e"]
Sep 14 11:32:22.227269 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:32:22.227245    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe608ddd_4bde_4706_b941_2ea209c0db52.slice/crio-b267bb1d2e69155e265e67f1f9b7540ff77d355875c62e7839bb4e4debda48d9 WatchSource:0}: Error finding container b267bb1d2e69155e265e67f1f9b7540ff77d355875c62e7839bb4e4debda48d9: Status 404 returned error can't find the container with id b267bb1d2e69155e265e67f1f9b7540ff77d355875c62e7839bb4e4debda48d9
Sep 14 11:32:22.925455 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:22.925415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-729f9e" event={"ID":"be608ddd-4bde-4706-b941-2ea209c0db52","Type":"ContainerStarted","Data":"6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7"}
Sep 14 11:32:22.925916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:22.925466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-729f9e" event={"ID":"be608ddd-4bde-4706-b941-2ea209c0db52","Type":"ContainerStarted","Data":"b267bb1d2e69155e265e67f1f9b7540ff77d355875c62e7839bb4e4debda48d9"}
Sep 14 11:32:22.925916 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:22.925485    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-3f4451" podUID="1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" containerName="test-curl-7938-3f4451" containerID="cri-o://e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f" gracePeriod=2
Sep 14 11:32:22.942305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:22.942244    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-729f9e" podStartSLOduration=1.9422248469999999 podStartE2EDuration="1.942224847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:32:22.939831632 +0000 UTC m=+1891.562384847" watchObservedRunningTime="2026-09-14 11:32:22.942224847 +0000 UTC m=+1891.564778060"
Sep 14 11:32:23.531826 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:23.531789    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-729f9e"]
Sep 14 11:32:23.536004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:23.535972    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-729f9e"]
Sep 14 11:32:23.930578 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:32:23.930547    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-729f9e" container="test-curl-7938-729f9e"
Sep 14 11:32:24.136643 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:24.136607    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:32:24.937069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:24.937025    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-729f9e" podUID="be608ddd-4bde-4706-b941-2ea209c0db52" containerName="test-curl-7938-729f9e" containerID="cri-o://6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7" gracePeriod=2
Sep 14 11:32:25.106251 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.106220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:25.238013 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.237924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7m2q\" (UniqueName: \"kubernetes.io/projected/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9-kube-api-access-x7m2q\") pod \"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9\" (UID: \"1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9\") "
Sep 14 11:32:25.240014 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.239992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9-kube-api-access-x7m2q" (OuterVolumeSpecName: "kube-api-access-x7m2q") pod "1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" (UID: "1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9"). InnerVolumeSpecName "kube-api-access-x7m2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:32:25.338948 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.338911    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7m2q\" (UniqueName: \"kubernetes.io/projected/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9-kube-api-access-x7m2q\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:25.499501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.499410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:25.918903 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.918867    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" path="/var/lib/kubelet/pods/1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9/volumes"
Sep 14 11:32:25.943188 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.943160    2563 generic.go:356] "Generic (PLEG): container finished" podID="1fb3ed41-1f83-44bb-bf2f-cfaa120d41b9" containerID="e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f" exitCode=137
Sep 14 11:32:25.943562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.943198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-3f4451"
Sep 14 11:32:25.943562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.943249    2563 scope.go:117] "RemoveContainer" containerID="e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f"
Sep 14 11:32:25.952507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.952487    2563 scope.go:117] "RemoveContainer" containerID="e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f"
Sep 14 11:32:25.952759 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:32:25.952743    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f\": container with ID starting with e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f not found: ID does not exist" containerID="e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f"
Sep 14 11:32:25.952832 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:25.952768    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f"} err="failed to get container status \"e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f\": rpc error: code = NotFound desc = could not find container \"e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f\": container with ID starting with e0a44dcb3f21953c9e4e70eeb9da7598793d493e67ddca697dea8cb9e898e04f not found: ID does not exist"
Sep 14 11:32:26.436072 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:26.436034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"]
Sep 14 11:32:26.447779 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:26.447747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:27.111439 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.111413    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:27.254991 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.254907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqq8c\" (UniqueName: \"kubernetes.io/projected/be608ddd-4bde-4706-b941-2ea209c0db52-kube-api-access-qqq8c\") pod \"be608ddd-4bde-4706-b941-2ea209c0db52\" (UID: \"be608ddd-4bde-4706-b941-2ea209c0db52\") "
Sep 14 11:32:27.256996 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.256968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be608ddd-4bde-4706-b941-2ea209c0db52-kube-api-access-qqq8c" (OuterVolumeSpecName: "kube-api-access-qqq8c") pod "be608ddd-4bde-4706-b941-2ea209c0db52" (UID: "be608ddd-4bde-4706-b941-2ea209c0db52"). InnerVolumeSpecName "kube-api-access-qqq8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:32:27.356101 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.356063    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq8c\" (UniqueName: \"kubernetes.io/projected/be608ddd-4bde-4706-b941-2ea209c0db52-kube-api-access-qqq8c\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:27.919050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.919014    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be608ddd-4bde-4706-b941-2ea209c0db52" path="/var/lib/kubelet/pods/be608ddd-4bde-4706-b941-2ea209c0db52/volumes"
Sep 14 11:32:27.957411 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.957363    2563 generic.go:356] "Generic (PLEG): container finished" podID="be608ddd-4bde-4706-b941-2ea209c0db52" containerID="6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7" exitCode=137
Sep 14 11:32:27.957561 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.957430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-729f9e"
Sep 14 11:32:27.957561 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.957457    2563 scope.go:117] "RemoveContainer" containerID="6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7"
Sep 14 11:32:27.966358 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.966339    2563 scope.go:117] "RemoveContainer" containerID="6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7"
Sep 14 11:32:27.966649 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:32:27.966630    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7\": container with ID starting with 6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7 not found: ID does not exist" containerID="6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7"
Sep 14 11:32:27.966701 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:27.966658    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7"} err="failed to get container status \"6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7\": rpc error: code = NotFound desc = could not find container \"6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7\": container with ID starting with 6d422d02bb41e2b6d5cbe750d033854672b06829a948b1d18bb7786af67beee7 not found: ID does not exist"
Sep 14 11:32:28.741965 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.741921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"]
Sep 14 11:32:28.742458 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.742313    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be608ddd-4bde-4706-b941-2ea209c0db52" containerName="test-curl-7938-729f9e"
Sep 14 11:32:28.742458 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.742324    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="be608ddd-4bde-4706-b941-2ea209c0db52" containerName="test-curl-7938-729f9e"
Sep 14 11:32:28.742458 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.742415    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="be608ddd-4bde-4706-b941-2ea209c0db52" containerName="test-curl-7938-729f9e"
Sep 14 11:32:28.746720 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.746693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.749651 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.749625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"default-dockercfg-tfffw\""
Sep 14 11:32:28.749834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.749665    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"kube-root-ca.crt\""
Sep 14 11:32:28.749834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.749807    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"rate-test-model-55c025-kserve-self-signed-certs\""
Sep 14 11:32:28.749949 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.749893    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-55c025\"/\"openshift-service-ca.crt\""
Sep 14 11:32:28.759342 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.759318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"]
Sep 14 11:32:28.780144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.780116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:28.869723 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.869680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-dshm\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.869723 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.869729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-model-cache\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.869960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.869753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-tmp-dir\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.869960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.869845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbmg\" (UniqueName: \"kubernetes.io/projected/3dd042d9-4fda-4545-99bc-749581a12350-kube-api-access-9mbmg\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.869960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.869879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd042d9-4fda-4545-99bc-749581a12350-tls-certs\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.869960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.869919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-home\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.970718 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.970689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbmg\" (UniqueName: \"kubernetes.io/projected/3dd042d9-4fda-4545-99bc-749581a12350-kube-api-access-9mbmg\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.970879 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.970724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd042d9-4fda-4545-99bc-749581a12350-tls-certs\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.970879 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.970854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-home\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.970979 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.970962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-dshm\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.971062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.970998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-model-cache\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.971062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.971033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-tmp-dir\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.971326 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.971307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-home\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.971439 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.971420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-model-cache\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.971499 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.971475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-tmp-dir\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.973220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.973193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-dshm\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.973624 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.973607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd042d9-4fda-4545-99bc-749581a12350-tls-certs\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:28.978685 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:28.978661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbmg\" (UniqueName: \"kubernetes.io/projected/3dd042d9-4fda-4545-99bc-749581a12350-kube-api-access-9mbmg\") pod \"rate-test-model-55c025-kserve-757ff9c447-tv7pv\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") " pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:29.056810 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:29.056717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:29.344612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:29.344584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"]
Sep 14 11:32:29.346752 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:32:29.346724    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dd042d9_4fda_4545_99bc_749581a12350.slice/crio-6b4e84dc68653369ccc2e8e861f01e53a6b1978a3fdc3b504c2aeeb6ba7fd06a WatchSource:0}: Error finding container 6b4e84dc68653369ccc2e8e861f01e53a6b1978a3fdc3b504c2aeeb6ba7fd06a: Status 404 returned error can't find the container with id 6b4e84dc68653369ccc2e8e861f01e53a6b1978a3fdc3b504c2aeeb6ba7fd06a
Sep 14 11:32:29.971510 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:29.971470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv" event={"ID":"3dd042d9-4fda-4545-99bc-749581a12350","Type":"ContainerStarted","Data":"fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82"}
Sep 14 11:32:29.971897 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:29.971519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv" event={"ID":"3dd042d9-4fda-4545-99bc-749581a12350","Type":"ContainerStarted","Data":"6b4e84dc68653369ccc2e8e861f01e53a6b1978a3fdc3b504c2aeeb6ba7fd06a"}
Sep 14 11:32:29.990456 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:29.990363    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv" podStartSLOduration=1.990341193 podStartE2EDuration="1.990341193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:32:29.989942444 +0000 UTC m=+1898.612495659" watchObservedRunningTime="2026-09-14 11:32:29.990341193 +0000 UTC m=+1898.612894407"
Sep 14 11:32:35.448954 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:35.448918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:32:38.515741 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:38.515702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:38.617122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:38.617090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:39.056812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:39.056778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:39.057006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:39.056829    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:39.070625 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:39.070597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:40.037813 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:40.037784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:32:41.455740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.455700    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb" podUID="a022f7fb-6e2b-4bcb-9807-194915530dea" containerName="main" containerID="cri-o://0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542" gracePeriod=45
Sep 14 11:32:41.706557 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.706501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:32:41.792524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjh4p\" (UniqueName: \"kubernetes.io/projected/a022f7fb-6e2b-4bcb-9807-194915530dea-kube-api-access-sjh4p\") pod \"a022f7fb-6e2b-4bcb-9807-194915530dea\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") "
Sep 14 11:32:41.792707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-home\") pod \"a022f7fb-6e2b-4bcb-9807-194915530dea\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") "
Sep 14 11:32:41.792707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-dshm\") pod \"a022f7fb-6e2b-4bcb-9807-194915530dea\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") "
Sep 14 11:32:41.792707 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-model-cache\") pod \"a022f7fb-6e2b-4bcb-9807-194915530dea\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") "
Sep 14 11:32:41.792880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a022f7fb-6e2b-4bcb-9807-194915530dea-tls-certs\") pod \"a022f7fb-6e2b-4bcb-9807-194915530dea\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") "
Sep 14 11:32:41.792880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-tmp-dir\") pod \"a022f7fb-6e2b-4bcb-9807-194915530dea\" (UID: \"a022f7fb-6e2b-4bcb-9807-194915530dea\") "
Sep 14 11:32:41.792975 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792862    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-home" (OuterVolumeSpecName: "home") pod "a022f7fb-6e2b-4bcb-9807-194915530dea" (UID: "a022f7fb-6e2b-4bcb-9807-194915530dea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:32:41.792975 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.792926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-model-cache" (OuterVolumeSpecName: "model-cache") pod "a022f7fb-6e2b-4bcb-9807-194915530dea" (UID: "a022f7fb-6e2b-4bcb-9807-194915530dea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:32:41.793055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.793042    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:41.793096 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.793061    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:41.793135 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.793098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a022f7fb-6e2b-4bcb-9807-194915530dea" (UID: "a022f7fb-6e2b-4bcb-9807-194915530dea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:32:41.794836 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.794811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-dshm" (OuterVolumeSpecName: "dshm") pod "a022f7fb-6e2b-4bcb-9807-194915530dea" (UID: "a022f7fb-6e2b-4bcb-9807-194915530dea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:32:41.794836 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.794823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a022f7fb-6e2b-4bcb-9807-194915530dea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a022f7fb-6e2b-4bcb-9807-194915530dea" (UID: "a022f7fb-6e2b-4bcb-9807-194915530dea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:32:41.795018 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.794837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a022f7fb-6e2b-4bcb-9807-194915530dea-kube-api-access-sjh4p" (OuterVolumeSpecName: "kube-api-access-sjh4p") pod "a022f7fb-6e2b-4bcb-9807-194915530dea" (UID: "a022f7fb-6e2b-4bcb-9807-194915530dea"). InnerVolumeSpecName "kube-api-access-sjh4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:32:41.893507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.893470    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjh4p\" (UniqueName: \"kubernetes.io/projected/a022f7fb-6e2b-4bcb-9807-194915530dea-kube-api-access-sjh4p\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:41.893507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.893500    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:41.893507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.893512    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a022f7fb-6e2b-4bcb-9807-194915530dea-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:41.893507 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:41.893521    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a022f7fb-6e2b-4bcb-9807-194915530dea-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:32:42.039169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.039074    2563 generic.go:356] "Generic (PLEG): container finished" podID="a022f7fb-6e2b-4bcb-9807-194915530dea" containerID="0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542" exitCode=0
Sep 14 11:32:42.039169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.039122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"
Sep 14 11:32:42.039169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.039122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb" event={"ID":"a022f7fb-6e2b-4bcb-9807-194915530dea","Type":"ContainerDied","Data":"0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542"}
Sep 14 11:32:42.039169 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.039160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb" event={"ID":"a022f7fb-6e2b-4bcb-9807-194915530dea","Type":"ContainerDied","Data":"f5d4e5038a55268ac8587dcb5a62ab81ca94ba1f8ad0839f930a7ff8162aea09"}
Sep 14 11:32:42.039505 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.039183    2563 scope.go:117] "RemoveContainer" containerID="0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542"
Sep 14 11:32:42.048612 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.048591    2563 scope.go:117] "RemoveContainer" containerID="0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542"
Sep 14 11:32:42.048884 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:32:42.048865    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542\": container with ID starting with 0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542 not found: ID does not exist" containerID="0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542"
Sep 14 11:32:42.048930 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.048894    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542"} err="failed to get container status \"0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542\": rpc error: code = NotFound desc = could not find container \"0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542\": container with ID starting with 0ae3b282f43ee94be624c2d56d5a4f572f73926dc25e9f5f1230a38a72ec0542 not found: ID does not exist"
Sep 14 11:32:42.054893 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.054865    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"]
Sep 14 11:32:42.060372 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:42.060350    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/sub-test-model-3f02a5-kserve-744ddf854f-p96cb"]
Sep 14 11:32:43.918791 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:43.918750    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a022f7fb-6e2b-4bcb-9807-194915530dea" path="/var/lib/kubelet/pods/a022f7fb-6e2b-4bcb-9807-194915530dea/volumes"
Sep 14 11:32:45.945628 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.945588    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"]
Sep 14 11:32:45.946264 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.946242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a022f7fb-6e2b-4bcb-9807-194915530dea" containerName="main"
Sep 14 11:32:45.946312 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.946269    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a022f7fb-6e2b-4bcb-9807-194915530dea" containerName="main"
Sep 14 11:32:45.946434 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.946419    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a022f7fb-6e2b-4bcb-9807-194915530dea" containerName="main"
Sep 14 11:32:45.951176 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.951154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:45.954539 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.954493    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"openshift-service-ca.crt\""
Sep 14 11:32:45.954670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.954597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"default-dockercfg-q828q\""
Sep 14 11:32:45.954887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.954513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"rate-test-model-3f02a5-kserve-self-signed-certs\""
Sep 14 11:32:45.954887 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.954802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f02a5\"/\"kube-root-ca.crt\""
Sep 14 11:32:45.957241 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.957196    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"]
Sep 14 11:32:45.977924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:45.977893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:32:46.024759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.024725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-dshm\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.024759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.024763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cb5s\" (UniqueName: \"kubernetes.io/projected/233f57a9-f325-4d03-9574-e8875a6288af-kube-api-access-6cb5s\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.025025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.024869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-tmp-dir\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.025025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.024924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-model-cache\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.025025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.024957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233f57a9-f325-4d03-9574-e8875a6288af-tls-certs\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.025025 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.024996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-home\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.125549 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.125516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-dshm\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.125549 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.125551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cb5s\" (UniqueName: \"kubernetes.io/projected/233f57a9-f325-4d03-9574-e8875a6288af-kube-api-access-6cb5s\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.125843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.125591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-tmp-dir\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.125843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.125613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-model-cache\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.125843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.125639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233f57a9-f325-4d03-9574-e8875a6288af-tls-certs\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.125843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.125661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-home\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.126069 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.126046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-model-cache\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.126132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.126077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-tmp-dir\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.126132 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.126104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-home\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.127902 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.127877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-dshm\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.128089 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.128072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233f57a9-f325-4d03-9574-e8875a6288af-tls-certs\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.134812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.134790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cb5s\" (UniqueName: \"kubernetes.io/projected/233f57a9-f325-4d03-9574-e8875a6288af-kube-api-access-6cb5s\") pod \"rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") " pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.262888 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.262801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:46.536409 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:46.536370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"]
Sep 14 11:32:46.538018 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:32:46.537986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233f57a9_f325_4d03_9574_e8875a6288af.slice/crio-bfd2a4e29cc1bda027d2c2d4d2480e205dc6f5d85f223324f3132802e3d4e8ec WatchSource:0}: Error finding container bfd2a4e29cc1bda027d2c2d4d2480e205dc6f5d85f223324f3132802e3d4e8ec: Status 404 returned error can't find the container with id bfd2a4e29cc1bda027d2c2d4d2480e205dc6f5d85f223324f3132802e3d4e8ec
Sep 14 11:32:47.070118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:47.070076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4" event={"ID":"233f57a9-f325-4d03-9574-e8875a6288af","Type":"ContainerStarted","Data":"118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261"}
Sep 14 11:32:47.070118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:47.070122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4" event={"ID":"233f57a9-f325-4d03-9574-e8875a6288af","Type":"ContainerStarted","Data":"bfd2a4e29cc1bda027d2c2d4d2480e205dc6f5d85f223324f3132802e3d4e8ec"}
Sep 14 11:32:47.106809 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:47.106757    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4" podStartSLOduration=2.106742524 podStartE2EDuration="2.106742524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:32:47.104759104 +0000 UTC m=+1915.727312318" watchObservedRunningTime="2026-09-14 11:32:47.106742524 +0000 UTC m=+1915.729295738"
Sep 14 11:32:49.639299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:49.639259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:32:56.263639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:56.263608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:56.264060 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:56.263657    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:56.275907 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:56.275878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:32:57.143327 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:32:57.143298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:33:03.652740 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:03.652697    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:33:12.934924 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:12.934882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:33:19.526156 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:19.526116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"]
Sep 14 11:33:19.550843 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:19.550806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:20.628969 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:20.628931    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"]
Sep 14 11:33:20.662235 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:20.662197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:21.077031 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:21.076937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:21.375751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:21.375718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:21.604290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:21.604246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:23.343600 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:23.343564    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:33:24.595295 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:24.595251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:24.768868 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:24.768822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:24.862816 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:24.862784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:25.345067 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:25.344967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:26.080064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:26.080034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:26.182207 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:26.182172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:26.293297 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:26.293261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:26.639300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:26.639268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:26.741785 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:26.741744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:31.649525 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:31.649488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:33:34.545522 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.545481    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv" podUID="3dd042d9-4fda-4545-99bc-749581a12350" containerName="main" containerID="cri-o://fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82" gracePeriod=45
Sep 14 11:33:34.802279 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.802223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:33:34.872670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-dshm\") pod \"3dd042d9-4fda-4545-99bc-749581a12350\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") "
Sep 14 11:33:34.872670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mbmg\" (UniqueName: \"kubernetes.io/projected/3dd042d9-4fda-4545-99bc-749581a12350-kube-api-access-9mbmg\") pod \"3dd042d9-4fda-4545-99bc-749581a12350\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") "
Sep 14 11:33:34.872915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-home\") pod \"3dd042d9-4fda-4545-99bc-749581a12350\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") "
Sep 14 11:33:34.872915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-tmp-dir\") pod \"3dd042d9-4fda-4545-99bc-749581a12350\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") "
Sep 14 11:33:34.872915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd042d9-4fda-4545-99bc-749581a12350-tls-certs\") pod \"3dd042d9-4fda-4545-99bc-749581a12350\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") "
Sep 14 11:33:34.872915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-model-cache\") pod \"3dd042d9-4fda-4545-99bc-749581a12350\" (UID: \"3dd042d9-4fda-4545-99bc-749581a12350\") "
Sep 14 11:33:34.873121 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-home" (OuterVolumeSpecName: "home") pod "3dd042d9-4fda-4545-99bc-749581a12350" (UID: "3dd042d9-4fda-4545-99bc-749581a12350"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:34.873121 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.872987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3dd042d9-4fda-4545-99bc-749581a12350" (UID: "3dd042d9-4fda-4545-99bc-749581a12350"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:34.873206 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.873135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-model-cache" (OuterVolumeSpecName: "model-cache") pod "3dd042d9-4fda-4545-99bc-749581a12350" (UID: "3dd042d9-4fda-4545-99bc-749581a12350"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:34.873250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.873226    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:34.873250 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.873242    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:34.873336 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.873256    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:34.874926 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.874898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd042d9-4fda-4545-99bc-749581a12350-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3dd042d9-4fda-4545-99bc-749581a12350" (UID: "3dd042d9-4fda-4545-99bc-749581a12350"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:33:34.875321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.875295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-dshm" (OuterVolumeSpecName: "dshm") pod "3dd042d9-4fda-4545-99bc-749581a12350" (UID: "3dd042d9-4fda-4545-99bc-749581a12350"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:34.875321 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.875301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd042d9-4fda-4545-99bc-749581a12350-kube-api-access-9mbmg" (OuterVolumeSpecName: "kube-api-access-9mbmg") pod "3dd042d9-4fda-4545-99bc-749581a12350" (UID: "3dd042d9-4fda-4545-99bc-749581a12350"). InnerVolumeSpecName "kube-api-access-9mbmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:33:34.974516 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.974482    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd042d9-4fda-4545-99bc-749581a12350-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:34.974516 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.974513    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd042d9-4fda-4545-99bc-749581a12350-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:34.974681 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:34.974527    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbmg\" (UniqueName: \"kubernetes.io/projected/3dd042d9-4fda-4545-99bc-749581a12350-kube-api-access-9mbmg\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:35.358664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.358629    2563 generic.go:356] "Generic (PLEG): container finished" podID="3dd042d9-4fda-4545-99bc-749581a12350" containerID="fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82" exitCode=0
Sep 14 11:33:35.358845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.358671    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"
Sep 14 11:33:35.358845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.358710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv" event={"ID":"3dd042d9-4fda-4545-99bc-749581a12350","Type":"ContainerDied","Data":"fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82"}
Sep 14 11:33:35.358845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.358751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv" event={"ID":"3dd042d9-4fda-4545-99bc-749581a12350","Type":"ContainerDied","Data":"6b4e84dc68653369ccc2e8e861f01e53a6b1978a3fdc3b504c2aeeb6ba7fd06a"}
Sep 14 11:33:35.358845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.358767    2563 scope.go:117] "RemoveContainer" containerID="fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82"
Sep 14 11:33:35.370243 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.370215    2563 scope.go:117] "RemoveContainer" containerID="fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82"
Sep 14 11:33:35.370709 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:33:35.370681    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82\": container with ID starting with fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82 not found: ID does not exist" containerID="fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82"
Sep 14 11:33:35.370814 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.370712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82"} err="failed to get container status \"fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82\": rpc error: code = NotFound desc = could not find container \"fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82\": container with ID starting with fe135de6690c19cdd514146f6902d3c1afe1566420c272e2353dd5fb5e315e82 not found: ID does not exist"
Sep 14 11:33:35.380518 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.380490    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"]
Sep 14 11:33:35.383614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.383592    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-55c025/rate-test-model-55c025-kserve-757ff9c447-tv7pv"]
Sep 14 11:33:35.648827 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.648738    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4" podUID="233f57a9-f325-4d03-9574-e8875a6288af" containerName="main" containerID="cri-o://118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261" gracePeriod=45
Sep 14 11:33:35.910333 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.910265    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:33:35.918360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.918332    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dd042d9-4fda-4545-99bc-749581a12350" path="/var/lib/kubelet/pods/3dd042d9-4fda-4545-99bc-749581a12350/volumes"
Sep 14 11:33:35.982960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.982916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cb5s\" (UniqueName: \"kubernetes.io/projected/233f57a9-f325-4d03-9574-e8875a6288af-kube-api-access-6cb5s\") pod \"233f57a9-f325-4d03-9574-e8875a6288af\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") "
Sep 14 11:33:35.982960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.982965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-model-cache\") pod \"233f57a9-f325-4d03-9574-e8875a6288af\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") "
Sep 14 11:33:35.983224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.982996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-home\") pod \"233f57a9-f325-4d03-9574-e8875a6288af\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") "
Sep 14 11:33:35.983224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-tmp-dir\") pod \"233f57a9-f325-4d03-9574-e8875a6288af\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") "
Sep 14 11:33:35.983224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233f57a9-f325-4d03-9574-e8875a6288af-tls-certs\") pod \"233f57a9-f325-4d03-9574-e8875a6288af\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") "
Sep 14 11:33:35.983224 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-dshm\") pod \"233f57a9-f325-4d03-9574-e8875a6288af\" (UID: \"233f57a9-f325-4d03-9574-e8875a6288af\") "
Sep 14 11:33:35.983446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-model-cache" (OuterVolumeSpecName: "model-cache") pod "233f57a9-f325-4d03-9574-e8875a6288af" (UID: "233f57a9-f325-4d03-9574-e8875a6288af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:35.983446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-home" (OuterVolumeSpecName: "home") pod "233f57a9-f325-4d03-9574-e8875a6288af" (UID: "233f57a9-f325-4d03-9574-e8875a6288af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:35.983446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983321    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "233f57a9-f325-4d03-9574-e8875a6288af" (UID: "233f57a9-f325-4d03-9574-e8875a6288af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:35.983446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983420    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:35.983446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983437    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:35.983446 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.983444    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:35.985341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.985317    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-dshm" (OuterVolumeSpecName: "dshm") pod "233f57a9-f325-4d03-9574-e8875a6288af" (UID: "233f57a9-f325-4d03-9574-e8875a6288af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:33:35.985341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.985333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233f57a9-f325-4d03-9574-e8875a6288af-kube-api-access-6cb5s" (OuterVolumeSpecName: "kube-api-access-6cb5s") pod "233f57a9-f325-4d03-9574-e8875a6288af" (UID: "233f57a9-f325-4d03-9574-e8875a6288af"). InnerVolumeSpecName "kube-api-access-6cb5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:33:35.985524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:35.985362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233f57a9-f325-4d03-9574-e8875a6288af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "233f57a9-f325-4d03-9574-e8875a6288af" (UID: "233f57a9-f325-4d03-9574-e8875a6288af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:33:36.084732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.084697    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233f57a9-f325-4d03-9574-e8875a6288af-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:36.084732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.084725    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233f57a9-f325-4d03-9574-e8875a6288af-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:36.084732 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.084734    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cb5s\" (UniqueName: \"kubernetes.io/projected/233f57a9-f325-4d03-9574-e8875a6288af-kube-api-access-6cb5s\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:33:36.366290 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.366249    2563 generic.go:356] "Generic (PLEG): container finished" podID="233f57a9-f325-4d03-9574-e8875a6288af" containerID="118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261" exitCode=0
Sep 14 11:33:36.366486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.366296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"
Sep 14 11:33:36.366486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.366332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4" event={"ID":"233f57a9-f325-4d03-9574-e8875a6288af","Type":"ContainerDied","Data":"118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261"}
Sep 14 11:33:36.366486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.366376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4" event={"ID":"233f57a9-f325-4d03-9574-e8875a6288af","Type":"ContainerDied","Data":"bfd2a4e29cc1bda027d2c2d4d2480e205dc6f5d85f223324f3132802e3d4e8ec"}
Sep 14 11:33:36.366486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.366406    2563 scope.go:117] "RemoveContainer" containerID="118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261"
Sep 14 11:33:36.379994 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.379961    2563 scope.go:117] "RemoveContainer" containerID="118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261"
Sep 14 11:33:36.380333 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:33:36.380304    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261\": container with ID starting with 118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261 not found: ID does not exist" containerID="118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261"
Sep 14 11:33:36.380486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.380344    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261"} err="failed to get container status \"118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261\": rpc error: code = NotFound desc = could not find container \"118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261\": container with ID starting with 118c397a3a36c3572caa2368fadb5f8f2640c751f75083b52501568a594f4261 not found: ID does not exist"
Sep 14 11:33:36.393530 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.393500    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"]
Sep 14 11:33:36.398817 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:36.398792    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f02a5/rate-test-model-3f02a5-kserve-56978f5cb7-v7mt4"]
Sep 14 11:33:37.441926 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:37.441893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:37.542166 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:37.542129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:37.918750 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:37.918715    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233f57a9-f325-4d03-9574-e8875a6288af" path="/var/lib/kubelet/pods/233f57a9-f325-4d03-9574-e8875a6288af/volumes"
Sep 14 11:33:48.986585 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:48.986491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:49.692748 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:49.692704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:49.795653 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:49.795621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:50.457238 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.457194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:50.710534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.710433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:50.733597 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.733564    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"]
Sep 14 11:33:50.734061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.734031    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233f57a9-f325-4d03-9574-e8875a6288af" containerName="main"
Sep 14 11:33:50.734061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.734053    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="233f57a9-f325-4d03-9574-e8875a6288af" containerName="main"
Sep 14 11:33:50.734209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.734070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd042d9-4fda-4545-99bc-749581a12350" containerName="main"
Sep 14 11:33:50.734209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.734079    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd042d9-4fda-4545-99bc-749581a12350" containerName="main"
Sep 14 11:33:50.734209 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.734203    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="233f57a9-f325-4d03-9574-e8875a6288af" containerName="main"
Sep 14 11:33:50.734363 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.734219    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dd042d9-4fda-4545-99bc-749581a12350" containerName="main"
Sep 14 11:33:50.738795 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.738773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.741080 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.741059    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d0f937-openshift-default-dockercfg-7nxv8\""
Sep 14 11:33:50.741265 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.741242    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 11:33:50.755832 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.755798    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"]
Sep 14 11:33:50.790544 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.790512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:50.815713 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.815675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-certs\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.815913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.815721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-podinfo\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.815913 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.815843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-socket\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.816039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.815913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-token\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.816039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.815960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-credential-socket\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.816039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.815998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-envoy\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.816039 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.816032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-data\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.816174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.816070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istiod-ca-cert\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.816174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.816143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4dp\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-kube-api-access-xj4dp\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-socket\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916568 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-token\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-credential-socket\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-envoy\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-data\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istiod-ca-cert\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.916824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.916756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4dp\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-kube-api-access-xj4dp\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-socket\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917090 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-credential-socket\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917198 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-certs\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917198 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-data\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917198 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-podinfo\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917445 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-certs\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.917538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.917516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istiod-ca-cert\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.919013 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.918992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-envoy\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.919429 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.919376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-podinfo\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.928226 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.928199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-token\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:50.928331 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:50.928287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4dp\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-kube-api-access-xj4dp\") pod \"e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") " pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:51.050004 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.049914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:51.164689 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.164654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:51.330648 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.330615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"]
Sep 14 11:33:51.331563 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:33:51.331531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5cee3e_9d3c_4d44_8091_fd46a7780595.slice/crio-3e68391a6f38d8e6559b00b63f47ef2858c6c5093205c9a9d487ba31251f9a54 WatchSource:0}: Error finding container 3e68391a6f38d8e6559b00b63f47ef2858c6c5093205c9a9d487ba31251f9a54: Status 404 returned error can't find the container with id 3e68391a6f38d8e6559b00b63f47ef2858c6c5093205c9a9d487ba31251f9a54
Sep 14 11:33:51.333804 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.333773    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:33:51.333880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.333838    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:33:51.333880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.333866    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:33:51.454257 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.454219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" event={"ID":"8b5cee3e-9d3c-4d44-8091-fd46a7780595","Type":"ContainerStarted","Data":"bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"}
Sep 14 11:33:51.454475 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.454272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" event={"ID":"8b5cee3e-9d3c-4d44-8091-fd46a7780595","Type":"ContainerStarted","Data":"3e68391a6f38d8e6559b00b63f47ef2858c6c5093205c9a9d487ba31251f9a54"}
Sep 14 11:33:51.474749 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.474680    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podStartSLOduration=1.4746582959999999 podStartE2EDuration="1.474658296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:51.473454266 +0000 UTC m=+1980.096007481" watchObservedRunningTime="2026-09-14 11:33:51.474658296 +0000 UTC m=+1980.097211513"
Sep 14 11:33:51.567257 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:51.567219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:52.050407 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:52.050364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:52.055822 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:52.055794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:52.462463 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:52.462429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:52.463369 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:52.463345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:33:52.791766 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:52.791671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:53.416292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:53.416228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:53.511422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:53.511360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:53.614444 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:53.614383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:54.305455 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:54.305406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:55.173649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:55.173613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:55.947764 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:55.947726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:56.048122 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:56.048084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:56.408595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:56.408557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:56.509658 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:56.509627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:56.576317 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:56.576283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:33:58.850884 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:33:58.850848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:34:08.504586 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:08.504538    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 11:34:08.505005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:08.504606    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 11:34:09.565093 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:09.565061    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerID="bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67" exitCode=0
Sep 14 11:34:09.565541 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:09.565101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" event={"ID":"8b5cee3e-9d3c-4d44-8091-fd46a7780595","Type":"ContainerDied","Data":"bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"}
Sep 14 11:34:09.565595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:09.565553    2563 scope.go:117] "RemoveContainer" containerID="bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"
Sep 14 11:34:09.566639 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:09.566612    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:34:09.566710 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:09.566680    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:34:09.566747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:09.566712    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 14 11:34:10.573920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:10.573878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" event={"ID":"8b5cee3e-9d3c-4d44-8091-fd46a7780595","Type":"ContainerStarted","Data":"6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde"}
Sep 14 11:34:11.051155 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:11.051115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:34:11.051589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:11.051495    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 11:34:11.051798 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:11.051772    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 11:34:12.050744 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:12.050699    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 11:34:12.051259 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:12.050769    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 11:34:13.050441 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:13.050375    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 11:34:13.050654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:13.050477    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 11:34:14.051026 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:14.050986    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 11:34:14.051648 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:14.051613    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 11:34:15.050347 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:15.050308    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 14 11:34:15.050665 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:15.050637    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 14 11:34:16.054038 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:16.054000    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:34:16.054548 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:16.054413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:34:16.055137 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:16.055117    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:34:18.006910 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:18.006875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:18.423100 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:18.423065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:18.670177 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:18.670138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:19.399684 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:19.399651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:20.173867 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:20.173834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:20.983974 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:20.983935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:27.518055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.518019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:27.846494 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.846380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"]
Sep 14 11:34:27.850058 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.850041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:27.852222 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.852196    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-df3763\"/\"kube-root-ca.crt\""
Sep 14 11:34:27.852830 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.852806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-df3763\"/\"default-dockercfg-bpxrl\""
Sep 14 11:34:27.852960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.852811    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-df3763\"/\"openshift-service-ca.crt\""
Sep 14 11:34:27.852960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.852851    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-df3763\"/\"ipp-route-df3763-kserve-self-signed-certs\""
Sep 14 11:34:27.860220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.860200    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"]
Sep 14 11:34:27.869378 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.869355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:27.982933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.982901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-model-cache\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:27.982933 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.982938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61716607-821d-49c0-807e-f3d0811d7d28-tls-certs\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:27.983210 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.983001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-tmp-dir\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:27.983210 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.983059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglfc\" (UniqueName: \"kubernetes.io/projected/61716607-821d-49c0-807e-f3d0811d7d28-kube-api-access-nglfc\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:27.983210 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.983113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-home\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:27.983210 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:27.983137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-dshm\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglfc\" (UniqueName: \"kubernetes.io/projected/61716607-821d-49c0-807e-f3d0811d7d28-kube-api-access-nglfc\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-home\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-dshm\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-model-cache\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084484 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61716607-821d-49c0-807e-f3d0811d7d28-tls-certs\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084715 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-tmp-dir\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084715 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-home\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084830 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-model-cache\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.084830 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.084826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-tmp-dir\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.086556 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.086531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-dshm\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.086826 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.086807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61716607-821d-49c0-807e-f3d0811d7d28-tls-certs\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.092247 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.092221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglfc\" (UniqueName: \"kubernetes.io/projected/61716607-821d-49c0-807e-f3d0811d7d28-kube-api-access-nglfc\") pod \"ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") " pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.161357 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.161329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:28.437997 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.437970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"]
Sep 14 11:34:28.439574 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:34:28.439544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61716607_821d_49c0_807e_f3d0811d7d28.slice/crio-6e9ed7807f5a2cd82eca6e01903396ba9cd75ba114a6d28df7c125b96c09cf04 WatchSource:0}: Error finding container 6e9ed7807f5a2cd82eca6e01903396ba9cd75ba114a6d28df7c125b96c09cf04: Status 404 returned error can't find the container with id 6e9ed7807f5a2cd82eca6e01903396ba9cd75ba114a6d28df7c125b96c09cf04
Sep 14 11:34:28.677243 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.677202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb" event={"ID":"61716607-821d-49c0-807e-f3d0811d7d28","Type":"ContainerStarted","Data":"d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68"}
Sep 14 11:34:28.677243 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.677249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb" event={"ID":"61716607-821d-49c0-807e-f3d0811d7d28","Type":"ContainerStarted","Data":"6e9ed7807f5a2cd82eca6e01903396ba9cd75ba114a6d28df7c125b96c09cf04"}
Sep 14 11:34:28.695921 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:28.695820    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb" podStartSLOduration=1.69580518 podStartE2EDuration="1.69580518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:34:28.694334036 +0000 UTC m=+2017.316887244" watchObservedRunningTime="2026-09-14 11:34:28.69580518 +0000 UTC m=+2017.318358394"
Sep 14 11:34:32.136866 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:32.136827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:34:38.162182 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:38.162148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:38.162670 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:38.162232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:38.175006 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:38.174983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:38.743285 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:38.743248    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:34:43.347118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:43.347081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:34:49.746651 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:49.746614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:50.078608 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:50.078500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:50.371292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:50.371256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:52.690328 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:52.690293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:53.287736 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:53.287694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:53.386230 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:53.386199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:53.580754 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:53.580660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:54.383142 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:54.383107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:55.190184 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:55.190142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:55.848034 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:55.847999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:56.489335 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.489304    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"]
Sep 14 11:34:56.492875 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.492857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.494981 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.494961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1ec353-serving-cert\""
Sep 14 11:34:56.505294 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.505273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"]
Sep 14 11:34:56.523465 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.523440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:56.546550 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.546525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.546655 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.546567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxcg\" (UniqueName: \"kubernetes.io/projected/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-kube-api-access-2fxcg\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.546655 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.546598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.546655 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.546647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.624681 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.624648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:56.647641 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.647617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.647738 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.647669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.647738 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.647702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxcg\" (UniqueName: \"kubernetes.io/projected/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-kube-api-access-2fxcg\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.647738 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.647720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.647888 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:34:56.647779    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-serving-cert: secret "maas-api-e2e-ipp-cleanup-1ec353-serving-cert" not found
Sep 14 11:34:56.647888 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:34:56.647853    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls podName:57205be8-7d20-4bd2-9500-f6d9b0f71b0d nodeName:}" failed. No retries permitted until 2026-09-14 11:34:57.147833437 +0000 UTC m=+2045.770386643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" (UID: "57205be8-7d20-4bd2-9500-f6d9b0f71b0d") : secret "maas-api-e2e-ipp-cleanup-1ec353-serving-cert" not found
Sep 14 11:34:56.648376 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.648354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.650215 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.650195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.656036 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.656012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxcg\" (UniqueName: \"kubernetes.io/projected/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-kube-api-access-2fxcg\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:56.913595 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:56.913558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:57.017510 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.017456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:57.097183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.097148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:34:57.151300 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.151273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:57.153662 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.153645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:57.406021 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.405997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:57.676885 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.676857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"]
Sep 14 11:34:57.678709 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:34:57.678677    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57205be8_7d20_4bd2_9500_f6d9b0f71b0d.slice/crio-26fcae28ad1ae5f622a1b7e0e886809c93824907ca05d3efe210ad117e21e31f WatchSource:0}: Error finding container 26fcae28ad1ae5f622a1b7e0e886809c93824907ca05d3efe210ad117e21e31f: Status 404 returned error can't find the container with id 26fcae28ad1ae5f622a1b7e0e886809c93824907ca05d3efe210ad117e21e31f
Sep 14 11:34:57.679943 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.679925    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:34:57.841002 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:57.840962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" event={"ID":"57205be8-7d20-4bd2-9500-f6d9b0f71b0d","Type":"ContainerStarted","Data":"26fcae28ad1ae5f622a1b7e0e886809c93824907ca05d3efe210ad117e21e31f"}
Sep 14 11:34:58.847991 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:58.847959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" event={"ID":"57205be8-7d20-4bd2-9500-f6d9b0f71b0d","Type":"ContainerStarted","Data":"795a7381946ac42c7b7942625a0cd29b848220f6b1abc3b21c1ea3baf44befc4"}
Sep 14 11:34:58.848420 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:58.848104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:34:58.864284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:34:58.864238    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" podStartSLOduration=2.500901107 podStartE2EDuration="2.864222846s" totalImagesPullingTime="363.321739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:34:58.863227464 +0000 UTC m=+2047.485780678" watchObservedRunningTime="2026-09-14 11:34:58.864222846 +0000 UTC m=+2047.486776060"
Sep 14 11:35:01.044223 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:01.044189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:35:04.859679 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:04.859647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:35:05.883832 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:05.883797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:17.367103 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:17.367022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:18.682897 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:18.682858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:19.381351 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:19.381317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:20.048199 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:20.048160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:20.799211 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:20.799170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:21.473818 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:21.473785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:22.296637 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:22.296599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:23.359711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:23.359677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:24.683934 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:24.683895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:25.470803 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:25.470767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:26.129574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:26.129540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:26.890101 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:26.890062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:27.639214 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:27.639174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:28.458751 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:28.458716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:34.912191 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:34.912155    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"]
Sep 14 11:35:34.913151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:34.912961    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" podUID="57205be8-7d20-4bd2-9500-f6d9b0f71b0d" containerName="maas-api" containerID="cri-o://795a7381946ac42c7b7942625a0cd29b848220f6b1abc3b21c1ea3baf44befc4" gracePeriod=30
Sep 14 11:35:34.947472 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:34.947438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:35.017459 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.017430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:35.065432 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.065383    2563 generic.go:356] "Generic (PLEG): container finished" podID="57205be8-7d20-4bd2-9500-f6d9b0f71b0d" containerID="795a7381946ac42c7b7942625a0cd29b848220f6b1abc3b21c1ea3baf44befc4" exitCode=0
Sep 14 11:35:35.065432 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.065418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" event={"ID":"57205be8-7d20-4bd2-9500-f6d9b0f71b0d","Type":"ContainerDied","Data":"795a7381946ac42c7b7942625a0cd29b848220f6b1abc3b21c1ea3baf44befc4"}
Sep 14 11:35:35.100062 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.100024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:35.153308 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.153284    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:35:35.160027 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.159998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:35.284976 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.284887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-metrics-certs\") pod \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") "
Sep 14 11:35:35.284976 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.284974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls\") pod \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") "
Sep 14 11:35:35.285149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.285033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fxcg\" (UniqueName: \"kubernetes.io/projected/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-kube-api-access-2fxcg\") pod \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") "
Sep 14 11:35:35.285149 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.285077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-openshift-service-ca\") pod \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\" (UID: \"57205be8-7d20-4bd2-9500-f6d9b0f71b0d\") "
Sep 14 11:35:35.285620 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.285563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57205be8-7d20-4bd2-9500-f6d9b0f71b0d" (UID: "57205be8-7d20-4bd2-9500-f6d9b0f71b0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:35:35.287257 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.287218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57205be8-7d20-4bd2-9500-f6d9b0f71b0d" (UID: "57205be8-7d20-4bd2-9500-f6d9b0f71b0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:35:35.287257 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.287237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57205be8-7d20-4bd2-9500-f6d9b0f71b0d" (UID: "57205be8-7d20-4bd2-9500-f6d9b0f71b0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:35:35.287469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.287340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-kube-api-access-2fxcg" (OuterVolumeSpecName: "kube-api-access-2fxcg") pod "57205be8-7d20-4bd2-9500-f6d9b0f71b0d" (UID: "57205be8-7d20-4bd2-9500-f6d9b0f71b0d"). InnerVolumeSpecName "kube-api-access-2fxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:35:35.386050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.386017    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:35:35.386050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.386044    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:35:35.386050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.386054    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fxcg\" (UniqueName: \"kubernetes.io/projected/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-kube-api-access-2fxcg\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:35:35.386309 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.386063    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57205be8-7d20-4bd2-9500-f6d9b0f71b0d-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:35:35.486029 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.485994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:35.588833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:35.588741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:36.073221 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:36.073187    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"
Sep 14 11:35:36.073221 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:36.073205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj" event={"ID":"57205be8-7d20-4bd2-9500-f6d9b0f71b0d","Type":"ContainerDied","Data":"26fcae28ad1ae5f622a1b7e0e886809c93824907ca05d3efe210ad117e21e31f"}
Sep 14 11:35:36.073730 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:36.073266    2563 scope.go:117] "RemoveContainer" containerID="795a7381946ac42c7b7942625a0cd29b848220f6b1abc3b21c1ea3baf44befc4"
Sep 14 11:35:36.089819 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:36.089796    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"]
Sep 14 11:35:36.093559 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:36.093537    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ec353-7c99f4b44c-n2tdj"]
Sep 14 11:35:36.378604 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:36.378566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:37.080620 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:37.080579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:37.728180 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:37.728143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:37.919205 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:37.919172    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57205be8-7d20-4bd2-9500-f6d9b0f71b0d" path="/var/lib/kubelet/pods/57205be8-7d20-4bd2-9500-f6d9b0f71b0d/volumes"
Sep 14 11:35:38.490772 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:38.490740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:38.934490 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:38.934450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:35:41.795291 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:41.795244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:41.895246 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:41.895205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:42.057376 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:42.057276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:45.795814 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:45.795777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:46.515433 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:46.515379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:47.310890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:47.310857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:48.011021 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:48.010989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:48.797932 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:48.797891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:49.484553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:49.484515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:35:52.062501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:52.062463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:35:52.080161 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:52.080139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:35:58.741103 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:35:58.741066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:36:01.054636 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.054598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:01.126469 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.126429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:01.185536 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.185503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:01.297230 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.297182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:01.396272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.396234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:01.634654 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.634603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:01.760379 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:01.760273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:02.488297 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:02.488255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:03.238762 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:03.238724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:04.102408 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:04.102353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:06.620545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:06.620511    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"]
Sep 14 11:36:06.669892 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:06.669852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:11.141379 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:11.141338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:36:21.640153 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:21.640090    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb" podUID="61716607-821d-49c0-807e-f3d0811d7d28" containerName="main" containerID="cri-o://d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68" gracePeriod=45
Sep 14 11:36:21.901349 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:21.901288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:36:22.031016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.030984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nglfc\" (UniqueName: \"kubernetes.io/projected/61716607-821d-49c0-807e-f3d0811d7d28-kube-api-access-nglfc\") pod \"61716607-821d-49c0-807e-f3d0811d7d28\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") "
Sep 14 11:36:22.031189 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-dshm\") pod \"61716607-821d-49c0-807e-f3d0811d7d28\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") "
Sep 14 11:36:22.031189 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-home\") pod \"61716607-821d-49c0-807e-f3d0811d7d28\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") "
Sep 14 11:36:22.031189 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-model-cache\") pod \"61716607-821d-49c0-807e-f3d0811d7d28\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") "
Sep 14 11:36:22.031189 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61716607-821d-49c0-807e-f3d0811d7d28-tls-certs\") pod \"61716607-821d-49c0-807e-f3d0811d7d28\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") "
Sep 14 11:36:22.031425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-tmp-dir\") pod \"61716607-821d-49c0-807e-f3d0811d7d28\" (UID: \"61716607-821d-49c0-807e-f3d0811d7d28\") "
Sep 14 11:36:22.031425 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031356    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-home" (OuterVolumeSpecName: "home") pod "61716607-821d-49c0-807e-f3d0811d7d28" (UID: "61716607-821d-49c0-807e-f3d0811d7d28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:36:22.031545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-model-cache" (OuterVolumeSpecName: "model-cache") pod "61716607-821d-49c0-807e-f3d0811d7d28" (UID: "61716607-821d-49c0-807e-f3d0811d7d28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:36:22.031664 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61716607-821d-49c0-807e-f3d0811d7d28" (UID: "61716607-821d-49c0-807e-f3d0811d7d28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:36:22.031997 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.031975    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:36:22.032068 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.032004    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:36:22.032068 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.032018    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:36:22.033485 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.033403    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-dshm" (OuterVolumeSpecName: "dshm") pod "61716607-821d-49c0-807e-f3d0811d7d28" (UID: "61716607-821d-49c0-807e-f3d0811d7d28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:36:22.033587 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.033512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61716607-821d-49c0-807e-f3d0811d7d28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61716607-821d-49c0-807e-f3d0811d7d28" (UID: "61716607-821d-49c0-807e-f3d0811d7d28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:36:22.033889 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.033873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61716607-821d-49c0-807e-f3d0811d7d28-kube-api-access-nglfc" (OuterVolumeSpecName: "kube-api-access-nglfc") pod "61716607-821d-49c0-807e-f3d0811d7d28" (UID: "61716607-821d-49c0-807e-f3d0811d7d28"). InnerVolumeSpecName "kube-api-access-nglfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:36:22.132844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.132801    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61716607-821d-49c0-807e-f3d0811d7d28-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:36:22.132844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.132838    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglfc\" (UniqueName: \"kubernetes.io/projected/61716607-821d-49c0-807e-f3d0811d7d28-kube-api-access-nglfc\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:36:22.132844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.132849    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61716607-821d-49c0-807e-f3d0811d7d28-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:36:22.337958 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.337854    2563 generic.go:356] "Generic (PLEG): container finished" podID="61716607-821d-49c0-807e-f3d0811d7d28" containerID="d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68" exitCode=0
Sep 14 11:36:22.337958 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.337896    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"
Sep 14 11:36:22.337958 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.337942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb" event={"ID":"61716607-821d-49c0-807e-f3d0811d7d28","Type":"ContainerDied","Data":"d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68"}
Sep 14 11:36:22.338243 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.337987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb" event={"ID":"61716607-821d-49c0-807e-f3d0811d7d28","Type":"ContainerDied","Data":"6e9ed7807f5a2cd82eca6e01903396ba9cd75ba114a6d28df7c125b96c09cf04"}
Sep 14 11:36:22.338243 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.338007    2563 scope.go:117] "RemoveContainer" containerID="d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68"
Sep 14 11:36:22.347574 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.347556    2563 scope.go:117] "RemoveContainer" containerID="d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68"
Sep 14 11:36:22.347870 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:36:22.347851    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68\": container with ID starting with d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68 not found: ID does not exist" containerID="d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68"
Sep 14 11:36:22.347937 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.347879    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68"} err="failed to get container status \"d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68\": rpc error: code = NotFound desc = could not find container \"d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68\": container with ID starting with d085dd4b844fd285b63c89360ac115746c7307e6622eac9115c319e64436fc68 not found: ID does not exist"
Sep 14 11:36:22.359801 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.359772    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"]
Sep 14 11:36:22.369844 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:22.368103    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-df3763/ipp-route-df3763-kserve-6bfd5b5dbc-qzhjb"]
Sep 14 11:36:23.918725 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:23.918690    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61716607-821d-49c0-807e-f3d0811d7d28" path="/var/lib/kubelet/pods/61716607-821d-49c0-807e-f3d0811d7d28/volumes"
Sep 14 11:36:35.799601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:35.799560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:43.524060 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:43.524027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:43.840980 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:43.840877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:49.399000 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:49.398892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:49.477931 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:49.477892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:49.836845 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:49.836729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:50.353757 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:50.353722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:51.289249 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:51.289208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:51.951553 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:51.951517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:52.703535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:52.703501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:36:53.457692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:36:53.457652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:00.594195 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:00.594141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:00.658653 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:00.658610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:00.759922 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:00.759880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:00.859360 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:00.859324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:01.139362 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:01.139239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:01.240144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:01.240104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:01.890915 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:01.890877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:02.711960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:02.711927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:03.739554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:03.739513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:37:06.995113 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:06.995078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:07.050438 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:07.050362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:07.297163 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:07.297077    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"]
Sep 14 11:37:07.297322 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:07.297287    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy" containerID="cri-o://6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde" gracePeriod=30
Sep 14 11:37:07.301414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:07.301372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:07.715078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:07.715041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:08.585383 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:08.585343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:09.214920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:09.214876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:09.988428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:09.988379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:12.545093 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.545069    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:37:12.620918 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.620883    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerID="6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde" exitCode=0
Sep 14 11:37:12.621108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.620926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" event={"ID":"8b5cee3e-9d3c-4d44-8091-fd46a7780595","Type":"ContainerDied","Data":"6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde"}
Sep 14 11:37:12.621108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.620975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq" event={"ID":"8b5cee3e-9d3c-4d44-8091-fd46a7780595","Type":"ContainerDied","Data":"3e68391a6f38d8e6559b00b63f47ef2858c6c5093205c9a9d487ba31251f9a54"}
Sep 14 11:37:12.621108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.621002    2563 scope.go:117] "RemoveContainer" containerID="6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde"
Sep 14 11:37:12.621108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.620933    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"
Sep 14 11:37:12.630473 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.630454    2563 scope.go:117] "RemoveContainer" containerID="bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"
Sep 14 11:37:12.639174 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.639154    2563 scope.go:117] "RemoveContainer" containerID="6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde"
Sep 14 11:37:12.639474 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:37:12.639450    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde\": container with ID starting with 6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde not found: ID does not exist" containerID="6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde"
Sep 14 11:37:12.639614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.639480    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde"} err="failed to get container status \"6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde\": rpc error: code = NotFound desc = could not find container \"6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde\": container with ID starting with 6dc565ef55ebb9fa22a426694306fcc47c04a3450368b63c4c89122d2fbfecde not found: ID does not exist"
Sep 14 11:37:12.639614 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.639499    2563 scope.go:117] "RemoveContainer" containerID="bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"
Sep 14 11:37:12.639743 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:37:12.639727    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67\": container with ID starting with bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67 not found: ID does not exist" containerID="bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"
Sep 14 11:37:12.639782 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.639748    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67"} err="failed to get container status \"bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67\": rpc error: code = NotFound desc = could not find container \"bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67\": container with ID starting with bafec5001ffce576058c8989e9bd84a6c5a4706b1a6958c4da6a75f7f8779f67 not found: ID does not exist"
Sep 14 11:37:12.678101 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-data\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj4dp\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-kube-api-access-xj4dp\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678136    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-certs\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-socket\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678229 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istiod-ca-cert\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-podinfo\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-envoy\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-token\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-credential-socket\") pod \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\" (UID: \"8b5cee3e-9d3c-4d44-8091-fd46a7780595\") "
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678490    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:37:12.678632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678760    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678870    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-data\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678889    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678899    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-workload-socket\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678908    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istiod-ca-cert\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.678960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.678916    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-credential-socket\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.680602 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.680581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:37:12.680701 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.680652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:37:12.680821 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.680799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-kube-api-access-xj4dp" (OuterVolumeSpecName: "kube-api-access-xj4dp") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "kube-api-access-xj4dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:37:12.680881 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.680793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b5cee3e-9d3c-4d44-8091-fd46a7780595" (UID: "8b5cee3e-9d3c-4d44-8091-fd46a7780595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 11:37:12.780219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.780196    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-podinfo\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.780219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.780219    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-envoy\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.780422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.780229    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-istio-token\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.780422 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.780237    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj4dp\" (UniqueName: \"kubernetes.io/projected/8b5cee3e-9d3c-4d44-8091-fd46a7780595-kube-api-access-xj4dp\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:12.943801 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.943773    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"]
Sep 14 11:37:12.948519 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:12.948496    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d0f937-openshift-default-57cf8b849d-zh8cq"]
Sep 14 11:37:13.919170 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:13.919138    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" path="/var/lib/kubelet/pods/8b5cee3e-9d3c-4d44-8091-fd46a7780595/volumes"
Sep 14 11:37:18.062993 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.062955    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"]
Sep 14 11:37:18.063399 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.063194    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" podUID="d072ca32-7a4a-4a11-8e2e-be109303b686" containerName="maas-api" containerID="cri-o://81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715" gracePeriod=30
Sep 14 11:37:18.099428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.099375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:18.206558 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.206511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:18.277908 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.277857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:18.321572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.321511    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:37:18.326099 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.326077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls\") pod \"d072ca32-7a4a-4a11-8e2e-be109303b686\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") "
Sep 14 11:37:18.326219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.326118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ca32-7a4a-4a11-8e2e-be109303b686-openshift-service-ca\") pod \"d072ca32-7a4a-4a11-8e2e-be109303b686\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") "
Sep 14 11:37:18.326219 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.326201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-metrics-certs\") pod \"d072ca32-7a4a-4a11-8e2e-be109303b686\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") "
Sep 14 11:37:18.326292 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.326242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24vsv\" (UniqueName: \"kubernetes.io/projected/d072ca32-7a4a-4a11-8e2e-be109303b686-kube-api-access-24vsv\") pod \"d072ca32-7a4a-4a11-8e2e-be109303b686\" (UID: \"d072ca32-7a4a-4a11-8e2e-be109303b686\") "
Sep 14 11:37:18.326535 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.326512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d072ca32-7a4a-4a11-8e2e-be109303b686-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d072ca32-7a4a-4a11-8e2e-be109303b686" (UID: "d072ca32-7a4a-4a11-8e2e-be109303b686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:37:18.328213 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.328184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d072ca32-7a4a-4a11-8e2e-be109303b686" (UID: "d072ca32-7a4a-4a11-8e2e-be109303b686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:37:18.328305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.328253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d072ca32-7a4a-4a11-8e2e-be109303b686-kube-api-access-24vsv" (OuterVolumeSpecName: "kube-api-access-24vsv") pod "d072ca32-7a4a-4a11-8e2e-be109303b686" (UID: "d072ca32-7a4a-4a11-8e2e-be109303b686"). InnerVolumeSpecName "kube-api-access-24vsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:37:18.328305 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.328272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d072ca32-7a4a-4a11-8e2e-be109303b686" (UID: "d072ca32-7a4a-4a11-8e2e-be109303b686"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:37:18.377202 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.377165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:18.427102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.427068    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:18.427102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.427097    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vsv\" (UniqueName: \"kubernetes.io/projected/d072ca32-7a4a-4a11-8e2e-be109303b686-kube-api-access-24vsv\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:18.427102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.427107    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d072ca32-7a4a-4a11-8e2e-be109303b686-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:18.427347 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.427117    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d072ca32-7a4a-4a11-8e2e-be109303b686-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:18.434087 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.434046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:18.656428 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.656368    2563 generic.go:356] "Generic (PLEG): container finished" podID="d072ca32-7a4a-4a11-8e2e-be109303b686" containerID="81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715" exitCode=0
Sep 14 11:37:18.656632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.656427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" event={"ID":"d072ca32-7a4a-4a11-8e2e-be109303b686","Type":"ContainerDied","Data":"81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715"}
Sep 14 11:37:18.656632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.656443    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"
Sep 14 11:37:18.656632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.656495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm" event={"ID":"d072ca32-7a4a-4a11-8e2e-be109303b686","Type":"ContainerDied","Data":"2de199dd47cba22ceec589e722805650805367169f65299e8bf60a2e3bc632f9"}
Sep 14 11:37:18.656632 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.656526    2563 scope.go:117] "RemoveContainer" containerID="81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715"
Sep 14 11:37:18.666747 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.666724    2563 scope.go:117] "RemoveContainer" containerID="81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715"
Sep 14 11:37:18.667053 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:37:18.667032    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715\": container with ID starting with 81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715 not found: ID does not exist" containerID="81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715"
Sep 14 11:37:18.667129 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.667061    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715"} err="failed to get container status \"81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715\": rpc error: code = NotFound desc = could not find container \"81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715\": container with ID starting with 81888ebe876d16c3624b98c686c1740547fb3f3ae1e2873d01a456d4cbd05715 not found: ID does not exist"
Sep 14 11:37:18.679701 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.679676    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"]
Sep 14 11:37:18.683490 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.683467    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-55c025-646f8d8485-k6vzm"]
Sep 14 11:37:18.757810 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.757746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:18.856678 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:18.856644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:19.599746 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:19.599705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:19.919439 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:19.919407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d072ca32-7a4a-4a11-8e2e-be109303b686" path="/var/lib/kubelet/pods/d072ca32-7a4a-4a11-8e2e-be109303b686/volumes"
Sep 14 11:37:20.847906 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:20.847868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:37:24.898218 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:24.898185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:24.956964 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:24.956925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:25.160835 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:25.160742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:25.627026 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:25.626986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:26.457676 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:26.457639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:27.121672 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:27.121635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:35.960532 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:35.960497    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"]
Sep 14 11:37:35.961047 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:35.960752    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" podUID="0e51aaea-b2cc-495b-bf46-212c2d8112d6" containerName="maas-api" containerID="cri-o://0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c" gracePeriod=30
Sep 14 11:37:35.988160 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:35.988123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.060939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.060903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.116855 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.116816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.173572 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.173538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.204745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.204718    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:37:36.232463 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.232378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.284335 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.284304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls\") pod \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") "
Sep 14 11:37:36.284524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.284364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8lb2\" (UniqueName: \"kubernetes.io/projected/0e51aaea-b2cc-495b-bf46-212c2d8112d6-kube-api-access-f8lb2\") pod \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") "
Sep 14 11:37:36.284524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.284424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-metrics-certs\") pod \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") "
Sep 14 11:37:36.284524 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.284448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e51aaea-b2cc-495b-bf46-212c2d8112d6-openshift-service-ca\") pod \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\" (UID: \"0e51aaea-b2cc-495b-bf46-212c2d8112d6\") "
Sep 14 11:37:36.284873 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.284849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e51aaea-b2cc-495b-bf46-212c2d8112d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e51aaea-b2cc-495b-bf46-212c2d8112d6" (UID: "0e51aaea-b2cc-495b-bf46-212c2d8112d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:37:36.286551 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.286524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e51aaea-b2cc-495b-bf46-212c2d8112d6-kube-api-access-f8lb2" (OuterVolumeSpecName: "kube-api-access-f8lb2") pod "0e51aaea-b2cc-495b-bf46-212c2d8112d6" (UID: "0e51aaea-b2cc-495b-bf46-212c2d8112d6"). InnerVolumeSpecName "kube-api-access-f8lb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:37:36.286551 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.286544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e51aaea-b2cc-495b-bf46-212c2d8112d6" (UID: "0e51aaea-b2cc-495b-bf46-212c2d8112d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:37:36.286711 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.286627    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e51aaea-b2cc-495b-bf46-212c2d8112d6" (UID: "0e51aaea-b2cc-495b-bf46-212c2d8112d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:37:36.385085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.385027    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-metrics-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:36.385085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.385077    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e51aaea-b2cc-495b-bf46-212c2d8112d6-openshift-service-ca\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:36.385085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.385089    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e51aaea-b2cc-495b-bf46-212c2d8112d6-maas-api-tls\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:36.385085 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.385097    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lb2\" (UniqueName: \"kubernetes.io/projected/0e51aaea-b2cc-495b-bf46-212c2d8112d6-kube-api-access-f8lb2\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:37:36.536067 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.535973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.636669 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.636613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:36.764253 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.764213    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e51aaea-b2cc-495b-bf46-212c2d8112d6" containerID="0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c" exitCode=0
Sep 14 11:37:36.764467 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.764268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" event={"ID":"0e51aaea-b2cc-495b-bf46-212c2d8112d6","Type":"ContainerDied","Data":"0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c"}
Sep 14 11:37:36.764467 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.764282    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"
Sep 14 11:37:36.764467 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.764319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk" event={"ID":"0e51aaea-b2cc-495b-bf46-212c2d8112d6","Type":"ContainerDied","Data":"9170f14e42bb21b2c595c4bc47fc7b92f7a03d3c85a773c84ef2aa82b2b90dfb"}
Sep 14 11:37:36.764467 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.764342    2563 scope.go:117] "RemoveContainer" containerID="0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c"
Sep 14 11:37:36.777794 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.777775    2563 scope.go:117] "RemoveContainer" containerID="0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c"
Sep 14 11:37:36.778038 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:37:36.778019    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c\": container with ID starting with 0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c not found: ID does not exist" containerID="0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c"
Sep 14 11:37:36.778095 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.778047    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c"} err="failed to get container status \"0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c\": rpc error: code = NotFound desc = could not find container \"0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c\": container with ID starting with 0d38a124197fc4f48ebf3f0dea576b3dc3330d73e0197e13d3b474f370b32f2c not found: ID does not exist"
Sep 14 11:37:36.788973 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.788913    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"]
Sep 14 11:37:36.792353 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:36.792331    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f02a5-85dc49bf57-dgxkk"]
Sep 14 11:37:37.918016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:37.917983    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e51aaea-b2cc-495b-bf46-212c2d8112d6" path="/var/lib/kubelet/pods/0e51aaea-b2cc-495b-bf46-212c2d8112d6/volumes"
Sep 14 11:37:38.645144 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:38.645110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:37:42.682893 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:42.682851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:42.738078 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:42.738051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:42.944829 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:42.944740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:47.537486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:47.537448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:37:55.771610 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:55.771570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:37:55.845055 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:37:55.845016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:38:24.284880 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:38:24.284795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:39:09.835056 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:09.835017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:39:10.454942 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:10.454908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:39:11.841554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:11.841522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:39:36.752760 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:36.752725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:39:44.842145 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:44.842060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:39:47.248035 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:47.247992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:39:47.305501 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:47.305471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:39:52.347423 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:39:52.346236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:40:00.142148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:00.142104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:40:13.448272 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:13.448235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:40:21.345920 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:21.345887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:40:24.352862 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:24.352828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:40:25.727033 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:25.727000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:40:52.100348 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:52.100313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:40:52.121495 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:52.121471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:40:55.756480 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:55.756446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:40:55.838373 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:40:55.838338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:41:49.340064 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:41:49.340027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:41:57.145228 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:41:57.145190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:42:02.868016 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:02.867976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:42:04.044545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.044503    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"]
Sep 14 11:42:04.045065 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57205be8-7d20-4bd2-9500-f6d9b0f71b0d" containerName="maas-api"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045067    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="57205be8-7d20-4bd2-9500-f6d9b0f71b0d" containerName="maas-api"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045093    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045117    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61716607-821d-49c0-807e-f3d0811d7d28" containerName="main"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045126    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61716607-821d-49c0-807e-f3d0811d7d28" containerName="main"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045143    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d072ca32-7a4a-4a11-8e2e-be109303b686" containerName="maas-api"
Sep 14 11:42:04.045157 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045151    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d072ca32-7a4a-4a11-8e2e-be109303b686" containerName="maas-api"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e51aaea-b2cc-495b-bf46-212c2d8112d6" containerName="maas-api"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045170    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e51aaea-b2cc-495b-bf46-212c2d8112d6" containerName="maas-api"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045282    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045296    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d072ca32-7a4a-4a11-8e2e-be109303b686" containerName="maas-api"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045305    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e51aaea-b2cc-495b-bf46-212c2d8112d6" containerName="maas-api"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045315    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045325    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61716607-821d-49c0-807e-f3d0811d7d28" containerName="main"
Sep 14 11:42:04.045538 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.045333    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="57205be8-7d20-4bd2-9500-f6d9b0f71b0d" containerName="maas-api"
Sep 14 11:42:04.048677 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.048655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.050860 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.050840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c35b58a6-kserve-self-signed-certs\""
Sep 14 11:42:04.058935 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.058910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"]
Sep 14 11:42:04.077857 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.077827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:42:04.088763 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.088738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-home\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.088884 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.088787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-tmp-dir\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.088884 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.088834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-model-cache\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.088971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.088881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm68\" (UniqueName: \"kubernetes.io/projected/16bdfd7d-dc02-4cb4-9582-155eedced656-kube-api-access-zbm68\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.088971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.088915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bdfd7d-dc02-4cb4-9582-155eedced656-tls-certs\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.088971 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.088941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-dshm\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.189874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.189839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-tmp-dir\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.189874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.189878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-model-cache\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190094 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.189905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm68\" (UniqueName: \"kubernetes.io/projected/16bdfd7d-dc02-4cb4-9582-155eedced656-kube-api-access-zbm68\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190094 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.189929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bdfd7d-dc02-4cb4-9582-155eedced656-tls-certs\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190094 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.189953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-dshm\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190094 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.189987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-home\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190341 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.190319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-model-cache\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190385 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.190336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-tmp-dir\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.190486 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.190385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-home\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.192318 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.192293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-dshm\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.192521 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.192502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bdfd7d-dc02-4cb4-9582-155eedced656-tls-certs\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.199007 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.198985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm68\" (UniqueName: \"kubernetes.io/projected/16bdfd7d-dc02-4cb4-9582-155eedced656-kube-api-access-zbm68\") pod \"e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") " pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.358820 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.358788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:04.636227 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:04.636195    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"]
Sep 14 11:42:04.638169 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:42:04.638140    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16bdfd7d_dc02_4cb4_9582_155eedced656.slice/crio-bf0ca2db617e07b938d06bc7f9f30db028b179b2816c3dfa8fcc6e13cd69e4d5 WatchSource:0}: Error finding container bf0ca2db617e07b938d06bc7f9f30db028b179b2816c3dfa8fcc6e13cd69e4d5: Status 404 returned error can't find the container with id bf0ca2db617e07b938d06bc7f9f30db028b179b2816c3dfa8fcc6e13cd69e4d5
Sep 14 11:42:05.222591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:05.222553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4" event={"ID":"16bdfd7d-dc02-4cb4-9582-155eedced656","Type":"ContainerStarted","Data":"0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0"}
Sep 14 11:42:05.222591 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:05.222599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4" event={"ID":"16bdfd7d-dc02-4cb4-9582-155eedced656","Type":"ContainerStarted","Data":"bf0ca2db617e07b938d06bc7f9f30db028b179b2816c3dfa8fcc6e13cd69e4d5"}
Sep 14 11:42:05.241813 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:05.241763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4" podStartSLOduration=1.241748888 podStartE2EDuration="1.241748888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:42:05.240154598 +0000 UTC m=+2473.862707804" watchObservedRunningTime="2026-09-14 11:42:05.241748888 +0000 UTC m=+2473.864302102"
Sep 14 11:42:11.149712 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:11.149670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:42:13.970605 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:13.970570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:42:14.359097 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:14.359070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:14.359258 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:14.359120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:14.372023 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:14.371991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:15.288534 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:15.288505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:17.488917 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:17.488882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:42:18.248542 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:18.248509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:42:32.671775 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:32.671739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:42:39.372060 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:39.372019    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"]
Sep 14 11:42:39.383383 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:39.383343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:42:41.385758 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:41.385705    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:42:41.504370 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:41.504335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:42:50.345912 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:50.345822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:42:54.391556 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.391510    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4" podUID="16bdfd7d-dc02-4cb4-9582-155eedced656" containerName="main" containerID="cri-o://0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0" gracePeriod=45
Sep 14 11:42:54.646833 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.646767    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:54.744600 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-home\") pod \"16bdfd7d-dc02-4cb4-9582-155eedced656\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") "
Sep 14 11:42:54.744600 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-dshm\") pod \"16bdfd7d-dc02-4cb4-9582-155eedced656\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") "
Sep 14 11:42:54.744853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-tmp-dir\") pod \"16bdfd7d-dc02-4cb4-9582-155eedced656\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") "
Sep 14 11:42:54.744853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bdfd7d-dc02-4cb4-9582-155eedced656-tls-certs\") pod \"16bdfd7d-dc02-4cb4-9582-155eedced656\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") "
Sep 14 11:42:54.744853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-model-cache\") pod \"16bdfd7d-dc02-4cb4-9582-155eedced656\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") "
Sep 14 11:42:54.744853 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbm68\" (UniqueName: \"kubernetes.io/projected/16bdfd7d-dc02-4cb4-9582-155eedced656-kube-api-access-zbm68\") pod \"16bdfd7d-dc02-4cb4-9582-155eedced656\" (UID: \"16bdfd7d-dc02-4cb4-9582-155eedced656\") "
Sep 14 11:42:54.745091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-home" (OuterVolumeSpecName: "home") pod "16bdfd7d-dc02-4cb4-9582-155eedced656" (UID: "16bdfd7d-dc02-4cb4-9582-155eedced656"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:42:54.745091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.744929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16bdfd7d-dc02-4cb4-9582-155eedced656" (UID: "16bdfd7d-dc02-4cb4-9582-155eedced656"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:42:54.745091 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.745006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-model-cache" (OuterVolumeSpecName: "model-cache") pod "16bdfd7d-dc02-4cb4-9582-155eedced656" (UID: "16bdfd7d-dc02-4cb4-9582-155eedced656"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:42:54.745220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.745193    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-tmp-dir\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:42:54.745220 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.745212    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-model-cache\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:42:54.745288 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.745224    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-home\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:42:54.747063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.747027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bdfd7d-dc02-4cb4-9582-155eedced656-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16bdfd7d-dc02-4cb4-9582-155eedced656" (UID: "16bdfd7d-dc02-4cb4-9582-155eedced656"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:42:54.747063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.747041    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-dshm" (OuterVolumeSpecName: "dshm") pod "16bdfd7d-dc02-4cb4-9582-155eedced656" (UID: "16bdfd7d-dc02-4cb4-9582-155eedced656"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:42:54.747208 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.747063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bdfd7d-dc02-4cb4-9582-155eedced656-kube-api-access-zbm68" (OuterVolumeSpecName: "kube-api-access-zbm68") pod "16bdfd7d-dc02-4cb4-9582-155eedced656" (UID: "16bdfd7d-dc02-4cb4-9582-155eedced656"). InnerVolumeSpecName "kube-api-access-zbm68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:42:54.846299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.846263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbm68\" (UniqueName: \"kubernetes.io/projected/16bdfd7d-dc02-4cb4-9582-155eedced656-kube-api-access-zbm68\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:42:54.846299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.846292    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16bdfd7d-dc02-4cb4-9582-155eedced656-dshm\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:42:54.846299 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:54.846302    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16bdfd7d-dc02-4cb4-9582-155eedced656-tls-certs\") on node \"ip-10-0-143-154.ec2.internal\" DevicePath \"\""
Sep 14 11:42:55.510759 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.510721    2563 generic.go:356] "Generic (PLEG): container finished" podID="16bdfd7d-dc02-4cb4-9582-155eedced656" containerID="0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0" exitCode=0
Sep 14 11:42:55.511183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.510777    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"
Sep 14 11:42:55.511183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.510803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4" event={"ID":"16bdfd7d-dc02-4cb4-9582-155eedced656","Type":"ContainerDied","Data":"0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0"}
Sep 14 11:42:55.511183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.510847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4" event={"ID":"16bdfd7d-dc02-4cb4-9582-155eedced656","Type":"ContainerDied","Data":"bf0ca2db617e07b938d06bc7f9f30db028b179b2816c3dfa8fcc6e13cd69e4d5"}
Sep 14 11:42:55.511183 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.510863    2563 scope.go:117] "RemoveContainer" containerID="0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0"
Sep 14 11:42:55.520891 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.520875    2563 scope.go:117] "RemoveContainer" containerID="0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0"
Sep 14 11:42:55.521150 ip-10-0-143-154 kubenswrapper[2563]: E0914 11:42:55.521129    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0\": container with ID starting with 0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0 not found: ID does not exist" containerID="0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0"
Sep 14 11:42:55.521193 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.521159    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0"} err="failed to get container status \"0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0\": rpc error: code = NotFound desc = could not find container \"0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0\": container with ID starting with 0af4fc2c465b88971fb1ad6844a757bb1255dece6ebd4e64cc545a7f81ca53a0 not found: ID does not exist"
Sep 14 11:42:55.539649 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.539618    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"]
Sep 14 11:42:55.545661 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.545637    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c35b58a6-kserve-5d79dcd759-9wvv4"]
Sep 14 11:42:55.919333 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:55.919299    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16bdfd7d-dc02-4cb4-9582-155eedced656" path="/var/lib/kubelet/pods/16bdfd7d-dc02-4cb4-9582-155eedced656/volumes"
Sep 14 11:42:57.252138 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:42:57.252098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:43:04.646358 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:04.646320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:43:12.244046 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.244002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:43:12.494699 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.494602    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"]
Sep 14 11:43:12.495003 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.494991    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy"
Sep 14 11:43:12.495048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.495005    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5cee3e-9d3c-4d44-8091-fd46a7780595" containerName="istio-proxy"
Sep 14 11:43:12.495048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.495027    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16bdfd7d-dc02-4cb4-9582-155eedced656" containerName="main"
Sep 14 11:43:12.495048 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.495034    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="16bdfd7d-dc02-4cb4-9582-155eedced656" containerName="main"
Sep 14 11:43:12.495141 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.495097    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="16bdfd7d-dc02-4cb4-9582-155eedced656" containerName="main"
Sep 14 11:43:12.498279 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.498252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.501108 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.501081    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:43:12.501226 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.501160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:43:12.501847 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.501827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 14 11:43:12.501939 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.501867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\""
Sep 14 11:43:12.502010 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.501962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:43:12.516365 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.516336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"]
Sep 14 11:43:12.616848 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.616813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916efdde-1f0c-4752-bc58-e1724c0317fb-openshift-service-ca\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.616848 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.616851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855cj\" (UniqueName: \"kubernetes.io/projected/916efdde-1f0c-4752-bc58-e1724c0317fb-kube-api-access-855cj\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.617061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.616877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916efdde-1f0c-4752-bc58-e1724c0317fb-maas-api-tls\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.617061 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.616972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916efdde-1f0c-4752-bc58-e1724c0317fb-metrics-certs\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.717589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.717554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855cj\" (UniqueName: \"kubernetes.io/projected/916efdde-1f0c-4752-bc58-e1724c0317fb-kube-api-access-855cj\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.717589 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.717603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916efdde-1f0c-4752-bc58-e1724c0317fb-maas-api-tls\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.717812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.717638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916efdde-1f0c-4752-bc58-e1724c0317fb-metrics-certs\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.717812 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.717717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916efdde-1f0c-4752-bc58-e1724c0317fb-openshift-service-ca\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.718598 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.718571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/916efdde-1f0c-4752-bc58-e1724c0317fb-openshift-service-ca\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.720210 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.720188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/916efdde-1f0c-4752-bc58-e1724c0317fb-maas-api-tls\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.720414 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.720371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916efdde-1f0c-4752-bc58-e1724c0317fb-metrics-certs\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.728423 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.728382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855cj\" (UniqueName: \"kubernetes.io/projected/916efdde-1f0c-4752-bc58-e1724c0317fb-kube-api-access-855cj\") pod \"maas-api-7749ddb494-jw9qs\" (UID: \"916efdde-1f0c-4752-bc58-e1724c0317fb\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:12.808768 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:12.808660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:13.085893 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:13.085861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"]
Sep 14 11:43:13.087822 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:43:13.087789    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916efdde_1f0c_4752_bc58_e1724c0317fb.slice/crio-8e07595b95d781add5efef2a129277fe3d23cb6f6efe1596b3d053d597eddf57 WatchSource:0}: Error finding container 8e07595b95d781add5efef2a129277fe3d23cb6f6efe1596b3d053d597eddf57: Status 404 returned error can't find the container with id 8e07595b95d781add5efef2a129277fe3d23cb6f6efe1596b3d053d597eddf57
Sep 14 11:43:13.089147 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:13.089123    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:43:13.618779 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:13.618743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs" event={"ID":"916efdde-1f0c-4752-bc58-e1724c0317fb","Type":"ContainerStarted","Data":"65c53a328ff15df1c9066d8495fc58ca19eeaea45ba4dbb2537248e7a0713da5"}
Sep 14 11:43:13.619151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:13.618798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs" event={"ID":"916efdde-1f0c-4752-bc58-e1724c0317fb","Type":"ContainerStarted","Data":"8e07595b95d781add5efef2a129277fe3d23cb6f6efe1596b3d053d597eddf57"}
Sep 14 11:43:13.619151 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:13.618834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:13.638547 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:13.638496    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs" podStartSLOduration=1.2182553409999999 podStartE2EDuration="1.638479535s" totalImagesPullingTime="420.224194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:43:13.63723122 +0000 UTC m=+2542.259784434" watchObservedRunningTime="2026-09-14 11:43:13.638479535 +0000 UTC m=+2542.261032749"
Sep 14 11:43:19.088077 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:19.088045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:43:19.630346 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:19.630317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7749ddb494-jw9qs"
Sep 14 11:43:24.874609 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:24.874569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:43:31.447241 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:31.447198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gxd6k"]
Sep 14 11:43:41.029298 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:41.029268    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-8xv8p_1bb88491-f609-4a24-81f5-d7b0809d79ba/copy-manifests/0.log"
Sep 14 11:43:41.037874 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:41.037848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-8xv8p_1bb88491-f609-4a24-81f5-d7b0809d79ba/manager/0.log"
Sep 14 11:43:41.664063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:41.664030    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7749ddb494-jw9qs_916efdde-1f0c-4752-bc58-e1724c0317fb/maas-api/0.log"
Sep 14 11:43:43.154326 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:43.154295    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sxr4b_99166d86-7ea2-476e-adfb-cd36e574dd94/manager/0.log"
Sep 14 11:43:43.273013 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:43.272986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mpr2j_13b3ece8-bb0a-4950-9af4-74e28a9bb928/kuadrant-console-plugin/0.log"
Sep 14 11:43:43.524890 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:43.524819    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gxd6k_08e1dfa2-16bc-4873-88d0-d64c40192fd2/limitador/0.log"
Sep 14 11:43:44.240244 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:44.240210    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-7bj84_c8005ec9-a549-4e2b-8582-715150557301/kube-auth-proxy/0.log"
Sep 14 11:43:45.332918 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:45.332883    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4plhf_6986b80e-1be1-44d4-98a4-ce60b159b1a2/main/0.log"
Sep 14 11:43:45.714148 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:45.714094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmkk2r_2d51c855-b610-48f8-95d1-b82cdc8b50a8/main/0.log"
Sep 14 11:43:51.886554 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:51.886513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:43:51.991515 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:51.991473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log" limit=1024
Sep 14 11:43:55.514905 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:55.514874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x7sw9_82f1808b-0670-4a82-8c87-39180882715e/global-pull-secret-syncer/0.log"
Sep 14 11:43:55.672521 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:55.672482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4lglg_422e099d-bace-421a-a944-8f29a52a5ff5/konnectivity-agent/0.log"
Sep 14 11:43:56.003714 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:56.003681    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-154.ec2.internal_ed43c03ec92d5b25a955c89d154a23b6/haproxy/0.log"
Sep 14 11:43:59.765140 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:59.765102    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sxr4b_99166d86-7ea2-476e-adfb-cd36e574dd94/manager/0.log"
Sep 14 11:43:59.886562 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:43:59.886530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mpr2j_13b3ece8-bb0a-4950-9af4-74e28a9bb928/kuadrant-console-plugin/0.log"
Sep 14 11:44:00.052492 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:00.052375    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gxd6k_08e1dfa2-16bc-4873-88d0-d64c40192fd2/limitador/0.log"
Sep 14 11:44:02.633979 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:02.633941    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-gl2m9_cc7758e7-57f1-4ab8-b7c0-bf6d75e2dc96/cluster-monitoring-operator/0.log"
Sep 14 11:44:02.819022 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:02.818918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zrc4g_8a797eb2-ec29-4d76-827b-19e913d423de/kube-state-metrics/0.log"
Sep 14 11:44:02.872692 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:02.872662    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zrc4g_8a797eb2-ec29-4d76-827b-19e913d423de/kube-rbac-proxy-main/0.log"
Sep 14 11:44:02.908494 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:02.908459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-zrc4g_8a797eb2-ec29-4d76-827b-19e913d423de/kube-rbac-proxy-self/0.log"
Sep 14 11:44:03.185304 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.185271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bn8j_ca11ed89-1292-43c6-84fe-cb2035d1f52c/node-exporter/0.log"
Sep 14 11:44:03.242445 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.242416    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bn8j_ca11ed89-1292-43c6-84fe-cb2035d1f52c/kube-rbac-proxy/0.log"
Sep 14 11:44:03.270835 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.270808    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bn8j_ca11ed89-1292-43c6-84fe-cb2035d1f52c/init-textfile/0.log"
Sep 14 11:44:03.475970 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.475877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-769cx_922ac942-96fd-4206-b6c6-25c61035427e/kube-rbac-proxy-main/0.log"
Sep 14 11:44:03.534289 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.534258    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-769cx_922ac942-96fd-4206-b6c6-25c61035427e/kube-rbac-proxy-self/0.log"
Sep 14 11:44:03.582116 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.582082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-769cx_922ac942-96fd-4206-b6c6-25c61035427e/openshift-state-metrics/0.log"
Sep 14 11:44:03.919453 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.919408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-km5rx_c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd/prometheus-operator/0.log"
Sep 14 11:44:03.952233 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.952205    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-km5rx_c8cdc9e5-5d03-486e-bc0c-0fccf2abeefd/kube-rbac-proxy/0.log"
Sep 14 11:44:03.996629 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:03.996597    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-k2wm5_ecf4d8dd-3cf4-4464-98ed-df18607beb2e/prometheus-operator-admission-webhook/0.log"
Sep 14 11:44:04.197509 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:04.197423    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d44f79b99-zz4j2_d11741f4-df64-483c-a3cf-16d014c71751/thanos-query/0.log"
Sep 14 11:44:04.267952 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:04.267921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d44f79b99-zz4j2_d11741f4-df64-483c-a3cf-16d014c71751/kube-rbac-proxy-web/0.log"
Sep 14 11:44:04.345764 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:04.345733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d44f79b99-zz4j2_d11741f4-df64-483c-a3cf-16d014c71751/kube-rbac-proxy/0.log"
Sep 14 11:44:04.408951 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:04.408922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d44f79b99-zz4j2_d11741f4-df64-483c-a3cf-16d014c71751/prom-label-proxy/0.log"
Sep 14 11:44:04.488686 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:04.488601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d44f79b99-zz4j2_d11741f4-df64-483c-a3cf-16d014c71751/kube-rbac-proxy-rules/0.log"
Sep 14 11:44:04.625424 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:04.625380    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d44f79b99-zz4j2_d11741f4-df64-483c-a3cf-16d014c71751/kube-rbac-proxy-metrics/0.log"
Sep 14 11:44:05.073267 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.073234    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"]
Sep 14 11:44:05.077194 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.077170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.083796 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.083550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxfzq\"/\"kube-root-ca.crt\""
Sep 14 11:44:05.084278 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.084255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxfzq\"/\"default-dockercfg-2g5tp\""
Sep 14 11:44:05.084418 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.084297    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxfzq\"/\"openshift-service-ca.crt\""
Sep 14 11:44:05.104635 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.104598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"]
Sep 14 11:44:05.192307 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.192262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-podres\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.192545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.192314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsc8\" (UniqueName: \"kubernetes.io/projected/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-kube-api-access-rmsc8\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.192545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.192376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-proc\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.192545 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.192458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-sys\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.192668 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.192539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-lib-modules\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293256 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-sys\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-lib-modules\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-podres\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsc8\" (UniqueName: \"kubernetes.io/projected/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-kube-api-access-rmsc8\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-proc\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-sys\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-lib-modules\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293496 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-podres\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.293824 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.293575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-proc\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.302234 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.302213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsc8\" (UniqueName: \"kubernetes.io/projected/b8ac3f11-c94a-4c03-b514-b660cabdc2d1-kube-api-access-rmsc8\") pod \"perf-node-gather-daemonset-hdlhj\" (UID: \"b8ac3f11-c94a-4c03-b514-b660cabdc2d1\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.387284 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.387246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.686487 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.686371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"]
Sep 14 11:44:05.689658 ip-10-0-143-154 kubenswrapper[2563]: W0914 11:44:05.689626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb8ac3f11_c94a_4c03_b514_b660cabdc2d1.slice/crio-f91d292aa98012479839e0d51b5f92d2f3a094814df88298d3bcf3e452effe92 WatchSource:0}: Error finding container f91d292aa98012479839e0d51b5f92d2f3a094814df88298d3bcf3e452effe92: Status 404 returned error can't find the container with id f91d292aa98012479839e0d51b5f92d2f3a094814df88298d3bcf3e452effe92
Sep 14 11:44:05.917162 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.917129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-25w8h_c8a10276-9416-4b01-b3e6-f17e67e44f04/networking-console-plugin/0.log"
Sep 14 11:44:05.919953 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.919928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj" event={"ID":"b8ac3f11-c94a-4c03-b514-b660cabdc2d1","Type":"ContainerStarted","Data":"437ad65a4cac74db5ac52cc99700a0b9a08304d04e8f25ede2e7271c7544a283"}
Sep 14 11:44:05.920063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.919986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:05.920063 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.920003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj" event={"ID":"b8ac3f11-c94a-4c03-b514-b660cabdc2d1","Type":"ContainerStarted","Data":"f91d292aa98012479839e0d51b5f92d2f3a094814df88298d3bcf3e452effe92"}
Sep 14 11:44:05.953577 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:05.953455    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj" podStartSLOduration=0.953439513 podStartE2EDuration="953.439513ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:44:05.950056422 +0000 UTC m=+2594.572609635" watchObservedRunningTime="2026-09-14 11:44:05.953439513 +0000 UTC m=+2594.575992768"
Sep 14 11:44:06.554302 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:06.554257    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/1.log"
Sep 14 11:44:06.563960 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:06.563929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hh9kh_6e805c86-98b1-4bc0-b457-b1100590d637/console-operator/2.log"
Sep 14 11:44:07.793579 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:07.793546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-6gwgh_f1d017a4-92d9-4e85-9d7e-bb8f64c2265a/volume-data-source-validator/0.log"
Sep 14 11:44:09.023050 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:09.023024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-745q5_449a4fc3-67e5-41ad-b69f-33551d650e3a/dns/0.log"
Sep 14 11:44:09.091097 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:09.091066    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-745q5_449a4fc3-67e5-41ad-b69f-33551d650e3a/kube-rbac-proxy/0.log"
Sep 14 11:44:09.387834 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:09.387799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d8dg5_2c60ef6d-94b5-4a01-afb1-99c1f6e99d41/dns-node-resolver/0.log"
Sep 14 11:44:10.348366 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:10.348335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7psjm_0a6d424e-e203-4fae-9ff5-db0982a2f37b/node-ca/0.log"
Sep 14 11:44:11.933225 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:11.933188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-hdlhj"
Sep 14 11:44:12.086005 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:12.085966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-7bj84_c8005ec9-a549-4e2b-8582-715150557301/kube-auth-proxy/0.log"
Sep 14 11:44:13.535601 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:13.535565    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9lhw2_02c8187f-5f7e-4860-a58f-65496fd89b31/serve-healthcheck-canary/0.log"
Sep 14 11:44:14.354736 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:14.354682    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hnnpq_cbefa077-d861-42b7-be43-c4c3b0bdfaba/insights-operator/0.log"
Sep 14 11:44:14.355502 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:14.355483    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hnnpq_cbefa077-d861-42b7-be43-c4c3b0bdfaba/insights-operator/1.log"
Sep 14 11:44:15.118297 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:15.118263    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zs5cm_b851c979-1608-4c20-ac00-d0b030e36757/kube-rbac-proxy/0.log"
Sep 14 11:44:15.301959 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:15.301858    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zs5cm_b851c979-1608-4c20-ac00-d0b030e36757/exporter/0.log"
Sep 14 11:44:15.342745 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:15.342708    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-7bj84_c8005ec9-a549-4e2b-8582-715150557301/kube-auth-proxy/0.log"
Sep 14 11:44:15.397509 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:15.397478    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zs5cm_b851c979-1608-4c20-ac00-d0b030e36757/extractor/0.log"
Sep 14 11:44:17.687356 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:17.687324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-7bj84_c8005ec9-a549-4e2b-8582-715150557301/kube-auth-proxy/0.log"
Sep 14 11:44:18.832102 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:18.832069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-8xv8p_1bb88491-f609-4a24-81f5-d7b0809d79ba/manager/0.log"
Sep 14 11:44:18.889118 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:18.889091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-8xv8p_1bb88491-f609-4a24-81f5-d7b0809d79ba/copy-manifests/0.log"
Sep 14 11:44:20.572517 ip-10-0-143-154 kubenswrapper[2563]: I0914 11:44:20.572482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69cb796bd9-znrrn_de64e813-065d-470a-afa6-9c55de53e4b0/manager/0.log"